Distributions
› Ubuntu
› Fedora
› CentOS
中文资源站
› 网易开源镜像站
islujw
V2EX  ›  Linux

Linux 批量下载 json 中的多个 url 的文件

  •  
  •   islujw · Aug 29, 2018 · 5353 views
    This topic created in 3004 days ago, the information mentioned may be changed or developed.

    现有一个 json 文件,希望下载其中所有的 “ url ” 的文件:

    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json

    如何在 Linux 上批量下载呢?使用 wget 一个个复制粘贴效率太低了。

    13 replies  •  2018-09-23 23:50:10 +08:00
    hanxiV2EX
    hanxiV2EX
    Aug 29, 2018 via Android
    1
    解析 json,对每个 url 执行 wget 命令
    exkernel
    exkernel
    Aug 29, 2018
    2
    sed 一下就出来了, 不会的话用[jq]( https://stedolan.github.io/jq/)
    omph
    omph
    Aug 29, 2018
    3
    优雅一点的,用程序解析 json 中的 url,然后下载
    暴力点的,正则出 url
    fireapp
    fireapp
    Aug 29, 2018 via iPhone
    4
    sed | awk | wget 搞定
    JHerschel
    JHerschel
    Aug 29, 2018
    5
    import json
    file = open('pathToFile')
    file_to_json = json.load(file)
    for a in file_to_json:
    for b in a['assets']:
    print(b['url'])

    wget 或者 requests 搞定。
    zerofiny
    zerofiny
    Aug 29, 2018
    6
    ```
    curl http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json | grep url | sed -e 's/"url"\ ://;s/\,//g' | awk '{print "curl -L -O "$1}' | sh
    ```
    UnknownR
    UnknownR
    Aug 29, 2018
    7
    作为 windows 党,推荐.net core 和 powershell,先复制 json 内容至剪切板
    [string]$a = get-clipboard
    $b = ConvertFrom-Json -InputObject $a
    $b.assets.url
    就拿到了所有 url,再当成数组一个个遍历,多线程用 wget 下载就行了

    [URL]
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_GL_G004_C010_v03_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D011_D009_SIGNCMP_v15_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A005_C009_v05_t9_6M.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_B005_C011_t9_6M_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_H004_C008_v10_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A009_C009_t9_6M_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_C002_C005_0818SC_001_v01_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_GL_G010_C006_v08_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LW_L001_C006_t9_6M_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D011_C010_v10_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/plate_G002_C002_BG_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_C007_C011_08244D_001_v01_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A006_C008_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D001_C001_v03_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_H004_C010_4k_v01_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_LA_A008_C004_ALT_v33_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D002_C003_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_C007_C004_0824AJ_001_v01_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D001_C005_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_HK_H004_C013_t9_6M_HB_tag0.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/comp_DB_D008_C010_v04_6Mbps.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b1-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b2-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b3-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b3-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b3-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b4-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b4-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b4-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b5-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b5-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b5-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b6-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b7-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b7-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b7-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-4.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b8-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b8-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b9-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b9-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b9-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-1.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-2.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b10-3.mov
    http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/b8-1.mov
    ps1aniuge
    ps1aniuge
    Aug 30, 2018
    8
    win linux 通用 powershell:
    ps1aniuge
    ps1aniuge
    Aug 30, 2018
    9
    $a = Invoke-RestMethod http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json |ConvertFrom-Json
    $a.assets.url

    和楼上的道理一样。只不过这个网址现在好像,啥命令都不行了,502 Bad Gateway 了。
    mingyun
    mingyun
    Sep 1, 2018
    10
    @hasdream 命令行玩的 6
    ps1aniuge
    ps1aniuge
    Sep 1, 2018
    11
    $url = 'http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json'
    $a = Invoke-RestMethod -Uri $url
    $a.assets.url

    测试通过

    我不知道网址内容是否变了,反正现在不需要搞 json 了。
    iwtbauh
    iwtbauh
    Sep 5, 2018 via Android
    12
    jshon 了解一下

    #!/bin/bash

    url=`jshon -e xxxx -e url1 -u`
    curl -s ...
    abcbc
    abcbc
    Sep 23, 2018
    13
    用 jq 来处理很简单:

    curl -L http://a1.phobos.apple.com/us/r1000/000/Features/atv/AutumnResources/videos/entries.json | jq -r '.[].assets | .[].url' | xargs -I {} curl -LO {}
    About   ·   Help   ·   Advertise   ·   Blog   ·   API   ·   FAQ   ·   Privacy   ·   Solana   ·   971 Online   Highest 6679   ·     Select Language
    创意工作者们的社区
    World is powered by solitude
    VERSION: 3.9.8.5 · 44ms · UTC 20:02 · PVG 04:02 · LAX 12:02 · JFK 15:02
    ♥ Do have faith in what you're doing.