Install Node.js

1. curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
2. sudo apt install -y nodejs

Verify installation of Node.js and npm:

node -v
npm -v

For Other OS

Install MongoDB

1. sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6
2. echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list
3. sudo apt-get update
4. sudo apt-get install -y mongodb-org

Verify:

sudo service mongod start
mongo

For Other OS

Seeing escape characters when pressing the arrow keys in python shell

On OS X, I have different problem.

When I using system python shell, the keys is no problem, but problem in virtualenv. I'd try to reinstall/upgrade virtualenv/readline and nothing fixed.

While I try to import readline in problem python shell, get this error message:

ImportError: dlopen(/Users/raptor/.virtualenvs/bottle/lib/python2.7/lib-dynload/readline.so, 2): > Library not loaded: /usr/local/opt/readline/lib/libreadline.6.dylib
Referenced from: /Users/raptor/.virtualenvs/bottle/lib/python2.7/lib-dynload/readline.so
Reason: image not found

Cause there is /usr/local/opt/readline/lib/libreadline.7.dylib but not libreadline.6.dylib, so I make a symbol link:

ln libreadline.7.dylib libreadline.6.dylib
Problem has been solved!

这个问题第一次被问到是 7年前，常问常新。

Caveats with inspecting the live browser DOM

Since Firefox add-ons operate on a live browser DOM, what you’ll actually see when inspecting the page source is not the original HTML, but a modified one after applying some browser clean up and executing Javascript code. Firefox, in particular, is known for adding elements to tables. Scrapy, on the other hand, does not modify the original page HTML, so you won’t be able to extract any data if you use in your XPath expressions.

Therefore, you should keep in mind the following things when working with Firefox and XPath:

• Disable Firefox Javascript while inspecting the DOM looking for XPaths to be used in Scrapy
• Never use full XPath paths, use relative and clever ones based on attributes (such as id, class, width, etc) or any identifying features like contains(@href, 'image').
• Never include elements in your XPath expressions unless you really know what you’re doing

在浏览器中检查DOM的注意事项

Firefox插件操作的是活动的浏览器DOM(live browser DOM)，这意味着当您检查网页源码的时候， 其已经不是原始的HTML，而是经过浏览器清理并执行一些Javascript代码后的结果。 Firefox是个典型的例子，其会在table中添加 元素。 而Scrapy相反，其并不修改原始的HTML，因此如果在XPath表达式中使用 ，您将获取不到任何数据。

所以，当XPath配合Firefox使用时您需要记住以下几点:

• 当检查DOM来查找Scrapy使用的XPath时，禁用Firefox的Javascrpit。
• 永远不要用完整的XPath路径。使用相对及基于属性(例如 id ， class ， width 等)的路径或者具有区别性的特性例如contains(@href, 'image') 。
• 永远不要在XPath表达式中加入 元素，除非您知道您在做什么

Actually,

1. Chrome 也会这样
2. 使用 .css() CSS 选择器也会有同样的问题

from: https://doc.scrapy.org/en/1.3/topics/firefox.html 中文文档

What Ports Are Blocked?

Modified on: Fri, May 1, 2015 at 9:52 am EST

We block several ports that are commonly abused for DDOS attacks:
TCP & UDP port 17 TCP & UDP port 19 TCP & UDP port 1900 These blocks are permanent, and cannot be removed.

Depending on your account, outbound SMTP (port 25) may also be blocked. This block can be removed by opening a support ticket.

from: https://www.vultr.com/docs/what-ports-are-blocked

$ mongo
> use admin
> db.createUser( { user: "<Enter a username>", pwd: "<Enter a secure password>", roles: [ { role: "readWriteAnyDatabase", db: "admin" }, { role: "userAdminAnyDatabase", db: "admin" } ] } )
> use <Enter a app name>
> db.createUser( { user: "<Enter a app username>", pwd: "<Enter a secure password>", roles: [ { role: "readWrite", db: "<Enter a app name>" }, { role: "clusterMonitor", db: "admin" } ] } )
> quit()

Enable database authorization in the MongoDB configuration file /usr/local/etc/mongod.conf by add security: and enabling authorization:

security:
  authorization: enabled

Restart MongoDB and verify the administrative user created earlier can connect:

$ brew services restart mongodb
$ mongo -u your_username -p your_password --authenticationDatabase=admin