id:rx7(@namikawa)の技術メモ このページをアンテナに追加 RSSフィード

過去記事一覧 | Main Blog (元RX-7乗りの適当な日々) | SBM (id:rx7のBookmark)


Google Custom Search API が 403 を返す時

| Google Custom Search API が 403 を返す時 - id:rx7(@namikawa)の技術メモ を含むブックマーク はてなブックマーク - Google Custom Search API が 403 を返す時 - id:rx7(@namikawa)の技術メモ Google Custom Search API が 403 を返す時 - id:rx7(@namikawa)の技術メモ のブックマークコメント


Custom Search API は制限をかけることができるのだけど、私はよく"IPアドレス"の制限をかけることが多く、


上記の制限で 403 Forbidden とステータスコードで怒られることがある。お恥ずかしい。

ちなみに、制限は Google Cloud Platform のコンソール画面から変更でき、以下の範囲で制限することが可能。

  • HTTPリファラー
  • IPアドレス
  • Androidアプリ
  • iOSアプリ
トラックバック - http://tech.g.hatena.ne.jp/rx7/20190331



| Dropboxで直接表示できる画像のURLリンクを生成する - id:rx7(@namikawa)の技術メモ を含むブックマーク はてなブックマーク - Dropboxで直接表示できる画像のURLリンクを生成する - id:rx7(@namikawa)の技術メモ Dropboxで直接表示できる画像のURLリンクを生成する - id:rx7(@namikawa)の技術メモ のブックマークコメント


いつの間にか Dropbox のサイト内で表示されるようになってしまっていたので、







  • "www.dropbox.com" を "dl.dropboxusercontent.com" に書き換え
  • "?dl=0" を削除




トラックバック - http://tech.g.hatena.ne.jp/rx7/20190131



| とあるサーバのbotアクセスランキング - id:rx7(@namikawa)の技術メモ を含むブックマーク はてなブックマーク - とあるサーバのbotアクセスランキング - id:rx7(@namikawa)の技術メモ とあるサーバのbotアクセスランキング - id:rx7(@namikawa)の技術メモ のブックマークコメント


$ cat /var/log/nginx/*-access.log.1 | cut -d " " -f12- | cut -d "\"" -f2 | egrep "bot|Bot|BOT" | sort | uniq -c | sort -nr

 108890 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  34594 Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)
  21757 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  15517 Mozilla/5.0 (compatible; SemrushBot/1.1~bl; +http://www.semrush.com/bot.html)
  14120 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
   8595 Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)
   7962 Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
   6274 Mozilla/5.0 (compatible; AhrefsBot/5.1; +http://ahrefs.com/robot/)
   3656 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
   3633 AdsBot-Google (+http://www.google.com/adsbot.html)
   2183 Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
   2120 Mozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36
   2074 Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
   1577 Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)
   1075 Googlebot-Image/1.0
   1009 Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
    410 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
    394 Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)
    324 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
    231 Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
    199 Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
    176 Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
    169 Mozilla/5.0 (compatible; memoryBot/1.24.54 +http://internetmemory.org/en/)
     81 Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/)
     53 istellabot/t.1
     41 roboto
     39 Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)
     34 adcrawler robot/1.1
     26 Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)
     25 Wotbox/2.01 (+http://www.wotbox.com/bot/)
     22 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
     17 msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
     17 adbeat_bot
     13 SafeDNSBot (https://www.safedns.com/searchbot)
     13 Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)
     10 Twitterbot/1.0
     10 Mozilla/5.0 (compatible; (smrjbot))
      9 Mozilla/5.0 (compatible; Yeti/1.1; +http://help.naver.com/support/robots.html)
      7 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36
      6 Hubot/2.19.0
      5 Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
      5 Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)
      3 Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
      2 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
      2 Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)
      1 Slack-ImgProxy (+https://api.slack.com/robots)
      1 Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
      1 Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com)
      1 Googlebot/2.1 (+http://www.google.com/bot.html
      1 bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)
トラックバック - http://tech.g.hatena.ne.jp/rx7/20170131



| curlで取得できるレスポンスタイムの内訳 - id:rx7(@namikawa)の技術メモ を含むブックマーク はてなブックマーク - curlで取得できるレスポンスタイムの内訳 - id:rx7(@namikawa)の技術メモ curlで取得できるレスポンスタイムの内訳 - id:rx7(@namikawa)の技術メモ のブックマークコメント


Step 4: Print timing breakdown

But we still don’t see the timing breakdown of the HTTP request as promised. If we go through all the options supported by curl (http://curl.haxx.se/docs/manpage.html) we will see that it has the capability of returning timing information on the hostname lookup ( time_namelookup), the time it takes to connect to the remote host ( time_connect), the time it takes to start transferring data ( time_pretrasnfer), the total time ( total_time) and many others. In order to display this information on our console in a pretty and human readable format copy and paste the following command:

$ curl -L --output /dev/null --silent --show-error --write-out 'lookup:        %{time_namelookup}\nconnect:       %{time_connect}\nappconnect:    %{time_appconnect}\npretransfer:   %{time_pretransfer}\nredirect:      %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal:         %{time_total}\n' 'google.com'
 lookup:        0.036
 connect:       0.053
 appconnect:    0.000
 pretransfer:   0.053
 redirect:      0.105
 starttransfer: 0.118
 total:         0.253

appconnect is the time, in seconds, it took from the start until the SSL/SSH/etc. connect/handshake to the remote host was completed. In the example above it is zero because we don’t use the HTTPS protocol. If you try the same command with “https://google.com” you get the following nonzero value for appconnect:

$ curl -L –output /dev/null –silent –show-error –write-out ‘lookup:        %{time_namelookup}\nconnect:       %{time_connect}\nappconnect:    %{time_appconnect}\npretransfer:   %{time_pretransfer}\nredirect:      %{time_redirect}\nstarttransfer: %{time_starttransfer}\ntotal:         %{time_total}\n’ ‘https://google.com’
lookup:        0.038
connect:       0.063
appconnect:    0.177
pretransfer:   0.178
redirect:      0.225
starttransfer: 0.679
total:         0.929
HTTP Transaction Timing with Curl | NetBeez Network Monitoring


Here is what each value represents, according to curl’s manual page:

  • lookup: The time, in seconds, it took from the start until the name resolving was completed.
  • connect: The time, in seconds, it took from the start until the TCP connect to the remote host (or proxy) was completed.
  • appconnect: The time, in seconds, it took from the start until the SSL/SSH/etc connect/handshake to the remote host was completed. (Added in 7.19.0)
  • pretransfer: The time, in seconds, it took from the start until the file transfer was just about to begin. This includes all pre-transfer commands and negotiations that are specific to the particular protocol(s) involved.
  • redirect: The time, in seconds, it took for all redirection steps include name lookup, connect, pretransfer and transfer before the final transaction was started. time_redirect shows the complete execution time for multiple redirections. (Added in 7.12.3)
  • starttransfer: The time, in seconds, it took from the start until the first byte was just about to be transferred. This includes time_pretransfer and also the time the server needed to calculate the result.
  • total: The total time, in seconds, that the full operation lasted. The time will be displayed with millisecond resolution.
HTTP Transaction Timing with Curl | NetBeez Network Monitoring
トラックバック - http://tech.g.hatena.ne.jp/rx7/20161031



| シェルスクリプトが途中でTerminatedする笑い話 - id:rx7(@namikawa)の技術メモ を含むブックマーク はてなブックマーク - シェルスクリプトが途中でTerminatedする笑い話 - id:rx7(@namikawa)の技術メモ シェルスクリプトが途中でTerminatedする笑い話 - id:rx7(@namikawa)の技術メモ のブックマークコメント




"bash -x" で見てみると、どうも

pkill hogefuga


このシェルスクリプトのファイル名が "hogefuga_backup.sh" というファイル名だった!


トラックバック - http://tech.g.hatena.ne.jp/rx7/20160627