Install Node.js

  1. curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
  2. sudo apt install -y nodejs

Verify installation of Node.js and npm:

node -v
npm -v

For Other OS

Install MongoDB

  1. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6
  2. echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list
  3. sudo apt-get update
  4. sudo apt-get install -y mongodb-org

Verify:

sudo service mongod start
mongo

For Other OS

Seeing escape characters when pressing the arrow keys in python shell

On OS X, I have different problem.

When I using system python shell, the keys is no problem, but problem in virtualenv. I'd try to reinstall/upgrade virtualenv/readline and nothing fixed.

While I try to import readline in problem python shell, get this error message:

ImportError: dlopen(/Users/raptor/.virtualenvs/bottle/lib/python2.7/lib-dynload/readline.so, 2): > Library not loaded: /usr/local/opt/readline/lib/libreadline.6.dylib
Referenced from: /Users/raptor/.virtualenvs/bottle/lib/python2.7/lib-dynload/readline.so
Reason: image not found

Cause there is /usr/local/opt/readline/lib/libreadline.7.dylib but not libreadline.6.dylib, so I make a symbol link:

ln libreadline.7.dylib libreadline.6.dylib
Problem has been solved!

这个问题第一次被问到是 7年前,常问常新。

Caveats with inspecting the live browser DOM

Since Firefox add-ons operate on a live browser DOM, what you’ll actually see when inspecting the page source is not the original HTML, but a modified one after applying some browser clean up and executing Javascript code. Firefox, in particular, is known for adding <tbody> elements to tables. Scrapy, on the other hand, does not modify the original page HTML, so you won’t be able to extract any data if you use <tbody> in your XPath expressions.

Therefore, you should keep in mind the following things when working with Firefox and XPath:

  • Disable Firefox Javascript while inspecting the DOM looking for XPaths to be used in Scrapy
  • Never use full XPath paths, use relative and clever ones based on attributes (such as id, class, width, etc) or any identifying features like contains(@href, 'image').
  • Never include <tbody> elements in your XPath expressions unless you really know what you’re doing

在浏览器中检查DOM的注意事项

Firefox插件操作的是活动的浏览器DOM(live browser DOM),这意味着当您检查网页源码的时候, 其已经不是原始的HTML,而是经过浏览器清理并执行一些Javascript代码后的结果。 Firefox是个典型的例子,其会在table中添加 <tbody> 元素。 而Scrapy相反,其并不修改原始的HTML,因此如果在XPath表达式中使用 <tbody> ,您将获取不到任何数据。

所以,当XPath配合Firefox使用时您需要记住以下几点:

  • 当检查DOM来查找Scrapy使用的XPath时,禁用Firefox的Javascrpit。
  • 永远不要用完整的XPath路径。使用相对及基于属性(例如 id , class , width 等)的路径或者具有区别性的特性例如contains(@href, 'image') 。
  • 永远不要在XPath表达式中加入 <tbody> 元素,除非您知道您在做什么

Actually,

  1. Chrome 也会这样
  2. 使用 .css() CSS 选择器也会有同样的问题

from: https://doc.scrapy.org/en/1.3/topics/firefox.html 中文文档

What Ports Are Blocked?

Modified on: Fri, May 1, 2015 at 9:52 am EST

We block several ports that are commonly abused for DDOS attacks:
TCP & UDP port 17 TCP & UDP port 19 TCP & UDP port 1900 These blocks are permanent, and cannot be removed.

Depending on your account, outbound SMTP (port 25) may also be blocked. This block can be removed by opening a support ticket.

from: https://www.vultr.com/docs/what-ports-are-blocked

$ mongo
> use admin
> db.createUser( { user: "<Enter a username>", pwd: "<Enter a secure password>", roles: [ { role: "readWriteAnyDatabase", db: "admin" }, { role: "userAdminAnyDatabase", db: "admin" } ] } )
> use <Enter a app name>
> db.createUser( { user: "<Enter a app username>", pwd: "<Enter a secure password>", roles: [ { role: "readWrite", db: "<Enter a app name>" }, { role: "clusterMonitor", db: "admin" } ] } )
> quit()

Enable database authorization in the MongoDB configuration file /usr/local/etc/mongod.conf by add security: and enabling authorization:

security:
  authorization: enabled

Restart MongoDB and verify the administrative user created earlier can connect:

$ brew services restart mongodb
$ mongo -u your_username -p your_password --authenticationDatabase=admin