---
-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.04.19*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
-- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.04.19**
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2016.06.03*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2016.06.03**
### Before submitting an *issue* make sure you have:
- [ ] At least skimmed through [README](https://github.com/rg3/youtube-dl/blob/master/README.md) and **most notably** [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2016.04.19
+[debug] youtube-dl version 2016.06.03
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
*.mp4
*.m4a
*.m4v
+*.mp3
*.part
*.swp
test/testdata
+test/local_parameters.json
.tox
youtube-dl.zsh
+
+# IntelliJ related files
.idea
-.idea/*
+*.iml
+
+tmp/
- "3.4"
- "3.5"
sudo: false
+install:
+ - bash ./devscripts/install_srelay.sh
+ - export PATH=$PATH:$(pwd)/tmp/srelay-0.4.8b6
script: nosetests test --verbose
notifications:
email:
- filippo.valsorda@gmail.com
- - phihag@phihag.de
- yasoob.khld@gmail.com
# irc:
# channels:
Viťas Strádal
Kagami Hiiragi
Philip Huppert
+blahgeek
+Kevin Deldycke
+inondle
+Tomáš Čech
+Déstin Reed
all: youtube-dl README.md CONTRIBUTING.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish supportedsites
clean:
- rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish youtube_dl/extractor/lazy_extractors.py *.dump *.part *.info.json *.mp4 *.flv *.mp3 *.avi CONTRIBUTING.md.tmp ISSUE_TEMPLATE.md.tmp youtube-dl youtube-dl.exe
+ rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish youtube_dl/extractor/lazy_extractors.py *.dump *.part *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.jpg *.png CONTRIBUTING.md.tmp ISSUE_TEMPLATE.md.tmp youtube-dl youtube-dl.exe
find . -name "*.pyc" -delete
find . -name "*.class" -delete
ot: offlinetest
offlinetest: codetest
- $(PYTHON) -m nose --verbose test --exclude test_download.py --exclude test_age_restriction.py --exclude test_subtitles.py --exclude test_write_annotations.py --exclude test_youtube_lists.py --exclude test_iqiyi_sdk_interpreter.py
+ $(PYTHON) -m nose --verbose test --exclude test_download.py --exclude test_age_restriction.py --exclude test_subtitles.py --exclude test_write_annotations.py --exclude test_youtube_lists.py --exclude test_iqiyi_sdk_interpreter.py --exclude test_socks.py
tar: youtube-dl.tar.gz
pandoc -f markdown -t plain README.md -o README.txt
youtube-dl.1: README.md
- $(PYTHON) devscripts/prepare_manpage.py >youtube-dl.1.temp.md
+ $(PYTHON) devscripts/prepare_manpage.py youtube-dl.1.temp.md
pandoc -s -f markdown -t man youtube-dl.1.temp.md -o youtube-dl.1
rm -f youtube-dl.1.temp.md
To install it right away for all UNIX users (Linux, OS X, etc.), type:
- sudo curl https://yt-dl.org/latest/youtube-dl -o /usr/local/bin/youtube-dl
+ sudo curl -L https://yt-dl.org/latest/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl
If you do not have curl, you can alternatively use a recent wget:
sudo wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl
-Windows users can [download a .exe file](https://yt-dl.org/latest/youtube-dl.exe) and place it in their home directory or any other location on their [PATH](http://en.wikipedia.org/wiki/PATH_%28variable%29).
+Windows users can [download an .exe file](https://yt-dl.org/latest/youtube-dl.exe) and place it in any location on their [PATH](http://en.wikipedia.org/wiki/PATH_%28variable%29) except for `%SYSTEMROOT%\System32` (e.g. **do not** put in `C:\Windows\System32`).
-OS X users can install **youtube-dl** with [Homebrew](http://brew.sh/).
+You can also use pip:
+
+ sudo pip install --upgrade youtube-dl
+
+This command will update youtube-dl if you have already installed it. See the [pypi page](https://pypi.python.org/pypi/youtube_dl) for more information.
+
+OS X users can install youtube-dl with [Homebrew](http://brew.sh/):
brew install youtube-dl
-You can also use pip:
+Or with [MacPorts](https://www.macports.org/):
- sudo pip install youtube-dl
+ sudo port install youtube-dl
Alternatively, refer to the [developer instructions](#developer-instructions) for how to check out and work with the git repository. For further options, including PGP signatures, see the [youtube-dl Download Page](https://rg3.github.io/youtube-dl/download.html).
repairs broken URLs, but emits an error if
this is not possible instead of searching.
--ignore-config Do not read configuration files. When given
- in the global configuration file /etc
- /youtube-dl.conf: Do not read the user
+ in the global configuration file
+ /etc/youtube-dl.conf: Do not read the user
configuration in ~/.config/youtube-
dl/config (%APPDATA%/youtube-dl/config.txt
on Windows)
--no-color Do not emit color codes in output
## Network Options:
- --proxy URL Use the specified HTTP/HTTPS proxy. Pass in
- an empty string (--proxy "") for direct
- connection
+ --proxy URL Use the specified HTTP/HTTPS/SOCKS proxy.
+ To enable experimental SOCKS proxy, specify
+ a proper scheme. For example
+ socks5://127.0.0.1:1080/. Pass in an empty
+ string (--proxy "") for direct connection
--socket-timeout SECONDS Time to wait before giving up, in seconds
--source-address IP Client-side IP address to bind to
(experimental)
(experimental)
## Download Options:
- -r, --rate-limit LIMIT Maximum download rate in bytes per second
+ -r, --limit-rate RATE Maximum download rate in bytes per second
(e.g. 50K or 4.2M)
-R, --retries RETRIES Number of retries (default is 10), or
"infinite".
--xattr-set-filesize Set file xattribute ytdl.filesize with
expected filesize (experimental)
--hls-prefer-native Use the native HLS downloader instead of
- ffmpeg (experimental)
+ ffmpeg
+ --hls-prefer-ffmpeg Use ffmpeg instead of the native HLS
+ downloader
--hls-use-mpegts Use the mpegts container for HLS videos,
allowing to play the video while
downloading (some players may not be able
jar in
--cache-dir DIR Location in the filesystem where youtube-dl
can store some downloaded information
- permanently. By default $XDG_CACHE_HOME
- /youtube-dl or ~/.cache/youtube-dl . At the
- moment, only YouTube player files (for
- videos with obfuscated signatures) are
- cached, but that may change.
+ permanently. By default
+ $XDG_CACHE_HOME/youtube-dl or
+ ~/.cache/youtube-dl . At the moment, only
+ YouTube player files (for videos with
+ obfuscated signatures) are cached, but that
+ may change.
--no-cache-dir Disable filesystem caching
--rm-cache-dir Delete all filesystem cache files
# CONFIGURATION
-You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`.
+You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux and OS X, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`.
For example, with the following configuration file youtube-dl will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory:
```
### Authentication with `.netrc` file
-You may also want to configure automatic credentials storage for extractors that support authentication (by providing login and password with `--username` and `--password`) in order not to pass credentials as command line arguments on every youtube-dl execution and prevent tracking plain text passwords in the shell command history. You can achieve this using a [`.netrc` file](http://stackoverflow.com/tags/.netrc/info) on per extractor basis. For that you will need to create a`.netrc` file in your `$HOME` and restrict permissions to read/write by you only:
+You may also want to configure automatic credentials storage for extractors that support authentication (by providing login and password with `--username` and `--password`) in order not to pass credentials as command line arguments on every youtube-dl execution and prevent tracking plain text passwords in the shell command history. You can achieve this using a [`.netrc` file](http://stackoverflow.com/tags/.netrc/info) on per extractor basis. For that you will need to create a `.netrc` file in your `$HOME` and restrict permissions to read/write by you only:
```
touch $HOME/.netrc
chmod a-rwx,u+rw $HOME/.netrc
- `display_id`: An alternative identifier for the video
- `uploader`: Full name of the video uploader
- `license`: License name the video is licensed under
- - `creator`: The main artist who created the video
+ - `creator`: The creator of the video
- `release_date`: The date (YYYYMMDD) when the video was released
- `timestamp`: UNIX timestamp of the moment the video became available
- `upload_date`: Video upload date (YYYYMMDD)
- `episode_number`: Number of the video episode within a season
- `episode_id`: Id of the video episode
+Available for the media that is a track or a part of a music album:
+ - `track`: Title of the track
+ - `track_number`: Number of the track within an album or a disc
+ - `track_id`: Id of the track
+ - `artist`: Artist(s) of the track
+ - `genre`: Genre(s) of the track
+ - `album`: Title of the album the track belongs to
+ - `album_type`: Type of the album
+ - `album_artist`: List of all artists appeared on the album
+ - `disc_number`: Number of the disc or other physical medium the track belongs to
+ - `release_year`: Year (YYYY) when the album was released
+
Each aforementioned sequence when referenced in output template will be replaced by the actual value corresponding to the sequence name. Note that some of the sequences are not guaranteed to be present since they depend on the metadata obtained by particular extractor, such sequences will be replaced with `NA`.
For example for `-o %(title)s-%(id)s.%(ext)s` and mp4 video with title `youtube-dl test video` and id `BaW_jenozKcj` this will result in a `youtube-dl test video-BaW_jenozKcj.mp4` file created in the current directory.
Again, from then on you'll be able to update with `sudo youtube-dl -U`.
+### youtube-dl is extremely slow to start on Windows
+
+Add a file exclusion for `youtube-dl.exe` in Windows Defender settings.
+
### I'm getting an error `Unable to extract OpenGraph title` on YouTube playlists
YouTube changed their playlist format in March 2014 and later on, so you'll need at least youtube-dl 2014.07.25 to download all YouTube videos.
If you have installed youtube-dl with a package manager, pip, setup.py or a tarball, please use that to update. Note that Ubuntu packages do not seem to get updated anymore. Since we are not affiliated with Ubuntu, there is little we can do. Feel free to [report bugs](https://bugs.launchpad.net/ubuntu/+source/youtube-dl/+filebug) to the [Ubuntu packaging guys](mailto:ubuntu-motu@lists.ubuntu.com?subject=outdated%20version%20of%20youtube-dl) - all they have to do is update the package to a somewhat recent version. See above for a way to update.
+### I'm getting an error when trying to use output template: `error: using output template conflicts with using title, video ID or auto number`
+
+Make sure you are not using `-o` with any of these options `-t`, `--title`, `--id`, `-A` or `--auto-number` set in command line or in a configuration file. Remove the latter if any.
+
### Do I always have to pass `-citw`?
By default, youtube-dl intends to have the best options (incidentally, if you have a convincing case that these should be different, [please file an issue where you explain that](https://yt-dl.org/bug)). Therefore, it is unnecessary and sometimes harmful to copy long option strings from webpages. In particular, the only option out of `-citw` that is regularly useful is `-i`.
### I have downloaded a video but how can I play it?
-Once the video is fully downloaded, use any video player, such as [vlc](http://www.videolan.org) or [mplayer](http://www.mplayerhq.hu/).
+Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](http://www.videolan.org) or [mplayer](http://www.mplayerhq.hu/).
### I extracted a video URL with `-g`, but it does not play on another machine / in my webbrowser.
Since June 2012 ([#342](https://github.com/rg3/youtube-dl/issues/342)) youtube-dl is packed as an executable zipfile, simply unzip it (might need renaming to `youtube-dl.zip` first on some systems) or clone the git repository, as laid out above. If you modify the code, you can run it by executing the `__main__.py` file. To recompile the executable, run `make youtube-dl`.
-### The exe throws a *Runtime error from Visual C++*
+### The exe throws an error due to missing `MSVCR100.dll`
-To run the exe you need to install first the [Microsoft Visual C++ 2008 Redistributable Package](http://www.microsoft.com/en-us/download/details.aspx?id=29).
+To run the exe you need to install first the [Microsoft Visual C++ 2010 Redistributable Package (x86)](https://www.microsoft.com/en-US/download/details.aspx?id=5555).
### On Windows, how should I set up ffmpeg and youtube-dl? Where should I put the exe files?
If you want to find out whether a given URL is supported, simply call youtube-dl with it. If you get no videos back, chances are the URL is either not referring to a video or unsupported. You can find out which by examining the output (if you run youtube-dl on the console) or catching an `UnsupportedError` exception if you run it from a Python program.
+# Why do I need to go through that much red tape when filing bugs?
+
+Before we had the issue template, despite our extensive [bug reporting instructions](#bugs), about 80% of the issue reports we got were useless, for instance because people used ancient versions hundreds of releases old, because of simple syntactic errors (not in youtube-dl but in general shell usage), because the problem was alrady reported multiple times before, because people did not actually read an error message, even if it said "please install ffmpeg", because people did not mention the URL they were trying to download and many more simple, easy-to-avoid problems, many of whom were totally unrelated to youtube-dl.
+
+youtube-dl is an open-source project manned by too few volunteers, so we'd rather spend time fixing bugs where we are certain none of those simple problems apply, and where we can be reasonably confident to be able to reproduce the issue without asking the reporter repeatedly. As such, the output of `youtube-dl -v YOUR_URL_HERE` is really all that's required to file an issue. The issue template also guides you through some basic steps you can do, such as checking that your version of youtube-dl is current.
+
# DEVELOPER INSTRUCTIONS
Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
#!/usr/bin/python3
-from http.server import HTTPServer, BaseHTTPRequestHandler
-from socketserver import ThreadingMixIn
import argparse
import ctypes
import functools
+import shutil
+import subprocess
import sys
+import tempfile
import threading
import traceback
import os.path
+sys.path.insert(0, os.path.dirname(os.path.dirname((os.path.abspath(__file__)))))
+from youtube_dl.compat import (
+ compat_input,
+ compat_http_server,
+ compat_str,
+ compat_urlparse,
+)
+
+# These are not used outside of buildserver.py thus not in compat.py
+
+try:
+ import winreg as compat_winreg
+except ImportError: # Python 2
+ import _winreg as compat_winreg
-class BuildHTTPServer(ThreadingMixIn, HTTPServer):
+try:
+ import socketserver as compat_socketserver
+except ImportError: # Python 2
+ import SocketServer as compat_socketserver
+
+
+class BuildHTTPServer(compat_socketserver.ThreadingMixIn, compat_http_server.HTTPServer):
allow_reuse_address = True
action='store_const', dest='action', const='service',
help='Run as a Windows service')
parser.add_argument('-b', '--bind', metavar='<host:port>',
- action='store', default='localhost:8142',
+ action='store', default='0.0.0.0:8142',
help='Bind to host:port (default %default)')
options = parser.parse_args(args=args)
srv = BuildHTTPServer((host, port), BuildHTTPRequestHandler)
thr = threading.Thread(target=srv.serve_forever)
thr.start()
- input('Press ENTER to shut down')
+ compat_input('Press ENTER to shut down')
srv.shutdown()
thr.join()
os.remove(fname)
os.rmdir(path)
-#==============================================================================
-
class BuildError(Exception):
def __init__(self, output, code=500):
class PythonBuilder(object):
def __init__(self, **kwargs):
- pythonVersion = kwargs.pop('python', '2.7')
- try:
- key = _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, r'SOFTWARE\Python\PythonCore\%s\InstallPath' % pythonVersion)
+ python_version = kwargs.pop('python', '3.4')
+ python_path = None
+ for node in ('Wow6432Node\\', ''):
try:
- self.pythonPath, _ = _winreg.QueryValueEx(key, '')
- finally:
- _winreg.CloseKey(key)
- except Exception:
- raise BuildError('No such Python version: %s' % pythonVersion)
+ key = compat_winreg.OpenKey(
+ compat_winreg.HKEY_LOCAL_MACHINE,
+ r'SOFTWARE\%sPython\PythonCore\%s\InstallPath' % (node, python_version))
+ try:
+ python_path, _ = compat_winreg.QueryValueEx(key, '')
+ finally:
+ compat_winreg.CloseKey(key)
+ break
+ except Exception:
+ pass
+
+ if not python_path:
+ raise BuildError('No such Python version: %s' % python_version)
+
+ self.pythonPath = python_path
super(PythonBuilder, self).__init__(**kwargs)
def build(self):
try:
- subprocess.check_output([os.path.join(self.pythonPath, 'python.exe'), 'setup.py', 'py2exe'],
- cwd=self.buildPath)
+ proc = subprocess.Popen([os.path.join(self.pythonPath, 'python.exe'), 'setup.py', 'py2exe'], stdin=subprocess.PIPE, cwd=self.buildPath)
+ proc.wait()
+ #subprocess.check_output([os.path.join(self.pythonPath, 'python.exe'), 'setup.py', 'py2exe'],
+ # cwd=self.buildPath)
except subprocess.CalledProcessError as e:
raise BuildError(e.output)
pass
-class BuildHTTPRequestHandler(BaseHTTPRequestHandler):
+class BuildHTTPRequestHandler(compat_http_server.BaseHTTPRequestHandler):
actionDict = {'build': Builder, 'download': Builder} # They're the same, no more caching.
def do_GET(self):
- path = urlparse.urlparse(self.path)
- paramDict = dict([(key, value[0]) for key, value in urlparse.parse_qs(path.query).items()])
+ path = compat_urlparse.urlparse(self.path)
+ paramDict = dict([(key, value[0]) for key, value in compat_urlparse.parse_qs(path.query).items()])
action, _, path = path.path.strip('/').partition('/')
if path:
path = path.split('/')
builder.close()
except BuildError as e:
self.send_response(e.code)
- msg = unicode(e).encode('UTF-8')
+ msg = compat_str(e).encode('UTF-8')
self.send_header('Content-Type', 'text/plain; charset=UTF-8')
self.send_header('Content-Length', len(msg))
self.end_headers()
else:
self.send_response(500, 'Malformed URL')
-#==============================================================================
-
if __name__ == '__main__':
main()
--- /dev/null
+#!/usr/bin/env python
+from __future__ import unicode_literals
+
+import base64
+import json
+import mimetypes
+import netrc
+import optparse
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from youtube_dl.compat import (
+ compat_basestring,
+ compat_input,
+ compat_getpass,
+ compat_print,
+ compat_urllib_request,
+)
+from youtube_dl.utils import (
+ make_HTTPS_handler,
+ sanitized_Request,
+)
+
+
+class GitHubReleaser(object):
+ _API_URL = 'https://api.github.com/repos/rg3/youtube-dl/releases'
+ _UPLOADS_URL = 'https://uploads.github.com/repos/rg3/youtube-dl/releases/%s/assets?name=%s'
+ _NETRC_MACHINE = 'github.com'
+
+ def __init__(self, debuglevel=0):
+ self._init_github_account()
+ https_handler = make_HTTPS_handler({}, debuglevel=debuglevel)
+ self._opener = compat_urllib_request.build_opener(https_handler)
+
+ def _init_github_account(self):
+ try:
+ info = netrc.netrc().authenticators(self._NETRC_MACHINE)
+ if info is not None:
+ self._username = info[0]
+ self._password = info[2]
+ compat_print('Using GitHub credentials found in .netrc...')
+ return
+ else:
+ compat_print('No GitHub credentials found in .netrc')
+ except (IOError, netrc.NetrcParseError):
+ compat_print('Unable to parse .netrc')
+ self._username = compat_input(
+ 'Type your GitHub username or email address and press [Return]: ')
+ self._password = compat_getpass(
+ 'Type your GitHub password and press [Return]: ')
+
+ def _call(self, req):
+ if isinstance(req, compat_basestring):
+ req = sanitized_Request(req)
+ # Authorizing manually since GitHub does not response with 401 with
+ # WWW-Authenticate header set (see
+ # https://developer.github.com/v3/#basic-authentication)
+ b64 = base64.b64encode(
+ ('%s:%s' % (self._username, self._password)).encode('utf-8')).decode('ascii')
+ req.add_header('Authorization', 'Basic %s' % b64)
+ response = self._opener.open(req).read().decode('utf-8')
+ return json.loads(response)
+
+ def list_releases(self):
+ return self._call(self._API_URL)
+
+ def create_release(self, tag_name, name=None, body='', draft=False, prerelease=False):
+ data = {
+ 'tag_name': tag_name,
+ 'target_commitish': 'master',
+ 'name': name,
+ 'body': body,
+ 'draft': draft,
+ 'prerelease': prerelease,
+ }
+ req = sanitized_Request(self._API_URL, json.dumps(data).encode('utf-8'))
+ return self._call(req)
+
+ def create_asset(self, release_id, asset):
+ asset_name = os.path.basename(asset)
+ url = self._UPLOADS_URL % (release_id, asset_name)
+ # Our files are small enough to be loaded directly into memory.
+ data = open(asset, 'rb').read()
+ req = sanitized_Request(url, data)
+ mime_type, _ = mimetypes.guess_type(asset_name)
+ req.add_header('Content-Type', mime_type or 'application/octet-stream')
+ return self._call(req)
+
+
+def main():
+ parser = optparse.OptionParser(usage='%prog VERSION BUILDPATH')
+ options, args = parser.parse_args()
+ if len(args) != 2:
+ parser.error('Expected a version and a build directory')
+
+ version, build_path = args
+
+ releaser = GitHubReleaser()
+
+ new_release = releaser.create_release(version, name='youtube-dl %s' % version)
+ release_id = new_release['id']
+
+ for asset in os.listdir(build_path):
+ compat_print('Uploading %s...' % asset)
+ releaser.create_asset(release_id, os.path.join(build_path, asset))
+
+
+if __name__ == '__main__':
+ main()
--- /dev/null
+#!/bin/bash
+
+mkdir -p tmp && cd tmp
+wget -N http://downloads.sourceforge.net/project/socks-relay/socks-relay/srelay-0.4.8/srelay-0.4.8b6.tar.gz
+tar zxvf srelay-0.4.8b6.tar.gz
+cd srelay-0.4.8b6
+./configure
+make
from __future__ import unicode_literals
import io
+import optparse
import os.path
-import sys
import re
ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
README_FILE = os.path.join(ROOT_DIR, 'README.md')
+PREFIX = '''%YOUTUBE-DL(1)
+
+# NAME
+
+youtube\-dl \- download videos from youtube.com or other video platforms
+
+# SYNOPSIS
+
+**youtube-dl** \[OPTIONS\] URL [URL...]
+
+'''
+
+
+def main():
+ parser = optparse.OptionParser(usage='%prog OUTFILE.md')
+ options, args = parser.parse_args()
+ if len(args) != 1:
+ parser.error('Expected an output filename')
+
+ outfile, = args
+
+ with io.open(README_FILE, encoding='utf-8') as f:
+ readme = f.read()
+
+ readme = re.sub(r'(?s)^.*?(?=# DESCRIPTION)', '', readme)
+ readme = re.sub(r'\s+youtube-dl \[OPTIONS\] URL \[URL\.\.\.\]', '', readme)
+ readme = PREFIX + readme
+
+ readme = filter_options(readme)
+
+ with io.open(outfile, 'w', encoding='utf-8') as outf:
+ outf.write(readme)
+
def filter_options(readme):
ret = ''
return ret
-with io.open(README_FILE, encoding='utf-8') as f:
- readme = f.read()
-
-PREFIX = '''%YOUTUBE-DL(1)
-
-# NAME
-
-youtube\-dl \- download videos from youtube.com or other video platforms
-
-# SYNOPSIS
-
-**youtube-dl** \[OPTIONS\] URL [URL...]
-
-'''
-readme = re.sub(r'(?s)^.*?(?=# DESCRIPTION)', '', readme)
-readme = re.sub(r'\s+youtube-dl \[OPTIONS\] URL \[URL\.\.\.\]', '', readme)
-readme = PREFIX + readme
-
-readme = filter_options(readme)
-
-if sys.version_info < (3, 0):
- print(readme.encode('utf-8'))
-else:
- print(readme)
+if __name__ == '__main__':
+ main()
# * the git config user.signingkey is properly set
# You will need
-# pip install coverage nose rsa
+# pip install coverage nose rsa wheel
# TODO
# release notes
set -e
skip_tests=true
-if [ "$1" = '--run-tests' ]; then
- skip_tests=false
- shift
-fi
+buildserver='localhost:8142'
+
+while true
+do
+case "$1" in
+ --run-tests)
+ skip_tests=false
+ shift
+ ;;
+ --buildserver)
+ buildserver="$2"
+ shift 2
+ ;;
+ --*)
+ echo "ERROR: unknown option $1"
+ exit 1
+ ;;
+ *)
+ break
+ ;;
+esac
+done
if [ -z "$1" ]; then echo "ERROR: specify version number like this: $0 1994.09.06"; exit 1; fi
version="$1"
useless_files=$(find youtube_dl -type f -not -name '*.py')
if [ ! -z "$useless_files" ]; then echo "ERROR: Non-.py files in youtube_dl: $useless_files"; exit 1; fi
if [ ! -f "updates_key.pem" ]; then echo 'ERROR: updates_key.pem missing'; exit 1; fi
+if ! type pandoc >/dev/null 2>/dev/null; then echo 'ERROR: pandoc is missing'; exit 1; fi
+if ! python3 -c 'import rsa' 2>/dev/null; then echo 'ERROR: python3-rsa is missing'; exit 1; fi
+if ! python3 -c 'import wheel' 2>/dev/null; then echo 'ERROR: wheel is missing'; exit 1; fi
/bin/echo -e "\n### First of all, testing..."
make clean
REV=$(git rev-parse HEAD)
make youtube-dl youtube-dl.tar.gz
read -p "VM running? (y/n) " -n 1
-wget "http://localhost:8142/build/rg3/youtube-dl/youtube-dl.exe?rev=$REV" -O youtube-dl.exe
+wget "http://$buildserver/build/rg3/youtube-dl/youtube-dl.exe?rev=$REV" -O youtube-dl.exe
mkdir -p "build/$version"
mv youtube-dl youtube-dl.exe "build/$version"
mv youtube-dl.tar.gz "build/$version/youtube-dl-$version.tar.gz"
(cd build/$version/ && sha256sum $RELEASE_FILES > SHA2-256SUMS)
(cd build/$version/ && sha512sum $RELEASE_FILES > SHA2-512SUMS)
-/bin/echo -e "\n### Signing and uploading the new binaries to yt-dl.org ..."
+/bin/echo -e "\n### Signing and uploading the new binaries to GitHub..."
for f in $RELEASE_FILES; do gpg --passphrase-repeat 5 --detach-sig "build/$version/$f"; done
-scp -r "build/$version" ytdl@yt-dl.org:html/tmp/
-ssh ytdl@yt-dl.org "mv html/tmp/$version html/downloads/"
+
+ROOT=$(pwd)
+python devscripts/create-github-release.py $version "$ROOT/build/$version"
+
ssh ytdl@yt-dl.org "sh html/update_latest.sh $version"
/bin/echo -e "\n### Now switching to gh-pages..."
git clone --branch gh-pages --single-branch . build/gh-pages
-ROOT=$(pwd)
(
set -e
ORIGIN_URL=$(git config --get remote.origin.url)
- **22tracks:genre**
- **22tracks:track**
- **24video**
+ - **3qsdn**: 3Q SDN
- **3sat**
- **4tube**
- **56.com**
- **9gag**
- **abc.net.au**
- **Abc7News**
+ - **abcnews**
+ - **abcnews:video**
- **AcademicEarth:Course**
- **acast**
- **acast:channel**
- **arte.tv:future**
- **arte.tv:info**
- **arte.tv:magazine**
+ - **arte.tv:playlist**
- **AtresPlayer**
- **ATTTechChannel**
- **AudiMedia**
- **Bild**: Bild.de
- **BiliBili**
- **BioBioChileTV**
+ - **BIQLE**
- **BleacherReport**
- **BleacherReportCMS**
- **blinkx**
- **CBCPlayer**
- **CBS**
- **CBSInteractive**
+ - **CBSLocal**
- **CBSNews**: CBS News
- **CBSNewsLiveVideo**: CBS News Live Videos
- **CBSSports**
- **chirbit**
- **chirbit:profile**
- **Cinchcast**
- - **Cinemassacre**
- **Clipfish**
- **cliphunter**
- **ClipRs**
- **CNN**
- **CNNArticle**
- **CNNBlogs**
- - **CollegeHumor**
- **CollegeRama**
- **ComCarCoff**
- **ComedyCentral**
- **ComedyCentralShows**: The Daily Show / The Colbert Report
- **CondeNast**: Condé Nast media group: Allure, Architectural Digest, Ars Technica, Bon Appétit, Brides, Condé Nast, Condé Nast Traveler, Details, Epicurious, GQ, Glamour, Golf Digest, SELF, Teen Vogue, The New Yorker, Vanity Fair, Vogue, W Magazine, WIRED
+ - **Coub**
- **Cracked**
- **Crackle**
- **Criterion**
- **culturebox.francetvinfo.fr**
- **CultureUnplugged**
- **CWTV**
+ - **DailyMail**
- **dailymotion**
- **dailymotion:playlist**
- **dailymotion:user**
- **defense.gouv.fr**
- **democracynow**
- **DHM**: Filmarchiv - Deutsches Historisches Museum
+ - **DigitallySpeaking**
- **Digiteka**
- **Discovery**
- **Dotsub**
- **Dropbox**
- **DrTuber**
- **DRTV**
- - **Dump**
- **Dumpert**
- **dvtv**: http://video.aktualne.cz/
- **dw**
- **exfm**: ex.fm
- **ExpoTV**
- **ExtremeTube**
+ - **EyedoTV**
- **facebook**
- **faz.net**
- **fc2**
- **Flickr**
- **Folketinget**: Folketinget (ft.dk; Danish parliament)
- **FootyRoom**
+ - **Formula1**
- **FOX**
- **Foxgay**
- **FoxNews**: Fox News and Fox Business Video
- **la7.tv**
- **Laola1Tv**
- **Le**: 乐视网
+ - **Learnr**
- **Lecture2Go**
- **Lemonde**
- **LePlaylist**
- **LetvCloud**: 乐视云
- **Libsyn**
+ - **life**: Life.ru
- **life:embed**
- - **lifenews**: LIFE | NEWS
- **limelight**
- **limelight:channel**
- **limelight:channel_list**
+ - **LiTV**
- **LiveLeak**
- **livestream**
- **livestream:original**
- **LnkGo**
+ - **loc**: Library of Congress
+ - **LocalNews8**
- **LoveHomePorn**
- **lrt.lt**
- **lynda**: lynda.com videos
- **mailru**: Видео@Mail.Ru
- **MakersChannel**
- **MakerTV**
- - **Malemotion**
- **MatchTV**
- **MDR**: MDR.DE and KiKA
- **media.ccc.de**
- **metacafe**
- **Metacritic**
- **Mgoon**
+ - **MGTV**: 芒果TV
- **Minhateca**
- **MinistryGrid**
- **Minoto**
- **mtvservices:embedded**
- **MuenchenTV**: münchen.tv
- **MusicPlayOn**
- - **muzu.tv**
+ - **mva**: Microsoft Virtual Academy videos
+ - **mva:course**: Microsoft Virtual Academy courses
- **Mwave**
+ - **MwaveMeetGreet**
- **MySpace**
- **MySpace:album**
- **MySpass**
- **nfl.com**
- **nhl.com**
- **nhl.com:news**: NHL news
- - **nhl.com:videocenter**: NHL videocenter category
+ - **nhl.com:videocenter**
+ - **nhl.com:videocenter:category**: NHL videocenter category
- **nick.com**
- **niconico**: ニコニコ動画
- **NiconicoPlaylist**
- **Patreon**
- **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
- **pcmag**
- - **Periscope**: Periscope
+ - **People**
+ - **periscope**: Periscope
+ - **periscope:user**: Periscope user videos
- **PhilharmonieDeParis**: Philharmonie de Paris
- **phoenix.de**
- **Photobucket**
- **Pinkbike**
- **Pladform**
- - **PlanetaPlay**
- **play.fm**
- **played.to**
- **PlaysTV**
- **qqmusic:playlist**: QQ音乐 - 歌单
- **qqmusic:singer**: QQ音乐 - 歌手
- **qqmusic:toplist**: QQ音乐 - 排行榜
- - **QuickVid**
- **R7**
- **radio.de**
- **radiobremen**
+ - **radiocanada**
+ - **RadioCanadaAudioVideo**
- **radiofrance**
- **RadioJavan**
- **Rai**
- **RedTube**
- **RegioTV**
- **Restudy**
+ - **Reuters**
- **ReverbNation**
- - **Revision3**
+ - **revision**
+ - **revision3:embed**
- **RICE**
- **RingTV**
- **RottenTomatoes**
- **ScreencastOMatic**
- **ScreenJunkies**
- **ScreenwaveMedia**
+ - **Seeker**
- **SenateISVP**
+ - **SendtoNews**
- **ServingSys**
- **Sexu**
- - **SexyKarma**: Sexy Karma and Watch Indian Porn
- **Shahid**
- **Shared**: shared.sx and vivo.sx
- **ShareSix**
- **smotri:broadcast**: Smotri.com broadcasts
- **smotri:community**: Smotri.com community videos
- **smotri:user**: Smotri.com user videos
- - **SnagFilms**
- - **SnagFilmsEmbed**
- **Snotr**
- **Sohu**
- **soundcloud**
- **Syfy**
- **SztvHu**
- **Tagesschau**
+ - **tagesschau:player**
- **Tapely**
- **Tass**
- **TDSLifeway**
- **TVCArticle**
- **tvigle**: Интернет-телевидение Tvigle.ru
- **tvland.com**
- - **tvp.pl**
- - **tvp.pl:Series**
+ - **tvp**: Telewizja Polska
+ - **tvp:series**
- **TVPlay**: TV3Play and related services
- **Tweakers**
- - **twitch:bookmarks**
- **twitch:chapter**
- **twitch:past_broadcasts**
- **twitch:profile**
- **USAToday**
- **ustream**
- **ustream:channel**
- - **Ustudio**
+ - **ustudio**
+ - **ustudio:embed**
- **Varzesh3**
- **Vbox7**
- **VeeHD**
- **Vessel**
- **Vesti**: Вести.Ru
- **Vevo**
+ - **VevoPlaylist**
- **VGTV**: VGTV, BTTV, FTV, Aftenposten and Aftonbladet
- **vh1.com**
- **Vice**
- **Vidzi**
- **vier**
- **vier:videos**
+ - **ViewLift**
+ - **ViewLiftEmbed**
- **Viewster**
- **Viidea**
- **viki**
- **VuClip**
- **vulture.com**
- **Walla**
- - **WashingtonPost**
+ - **washingtonpost**
+ - **washingtonpost:article**
- **wat.tv**
+ - **WatchIndianPorn**: Watch Indian Porn
- **WDR**
- **wdr:mobile**
- **WDRMaus**: Sendung mit der Maus
- **WSJ**: Wall Street Journal
- **XBef**
- **XboxClips**
- - **XFileShare**: XFileShare based sites: GorillaVid.in, daclips.in, movpod.in, fastvideo.in, realvid.net, filehoot.com and vidto.me
+ - **XFileShare**: XFileShare based sites: DaClips, FileHoot, GorillaVid, MovPod, PowerWatch, Rapidvideo.ws, TheVideoBee, Vidto, Streamin.To
- **XHamster**
- **XHamsterEmbed**
+ - **xiami:album**: 虾米音乐 - 专辑
+ - **xiami:artist**: 虾米音乐 - 歌手
+ - **xiami:collection**: 虾米音乐 - 精选集
+ - **xiami:song**: 虾米音乐
- **XMinus**
- **XNXX**
- **Xstream**
def get_params(override=None):
PARAMETERS_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
"parameters.json")
+ LOCAL_PARAMETERS_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
+ "local_parameters.json")
with io.open(PARAMETERS_FILE, encoding='utf-8') as pf:
parameters = json.load(pf)
+ if os.path.exists(LOCAL_PARAMETERS_FILE):
+ with io.open(LOCAL_PARAMETERS_FILE, encoding='utf-8') as pf:
+ parameters.update(json.load(pf))
if override:
parameters.update(override)
return parameters
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-from youtube_dl.utils import get_filesystem_encoding
from youtube_dl.compat import (
compat_getenv,
+ compat_setenv,
compat_etree_fromstring,
compat_expanduser,
compat_shlex_split,
compat_str,
+ compat_struct_unpack,
compat_urllib_parse_unquote,
compat_urllib_parse_unquote_plus,
compat_urllib_parse_urlencode,
class TestCompat(unittest.TestCase):
def test_compat_getenv(self):
test_str = 'тест'
- os.environ['YOUTUBE-DL-TEST'] = (
- test_str if sys.version_info >= (3, 0)
- else test_str.encode(get_filesystem_encoding()))
+ compat_setenv('YOUTUBE-DL-TEST', test_str)
self.assertEqual(compat_getenv('YOUTUBE-DL-TEST'), test_str)
+ def test_compat_setenv(self):
+ test_var = 'YOUTUBE-DL-TEST'
+ test_str = 'тест'
+ compat_setenv(test_var, test_str)
+ compat_getenv(test_var)
+ self.assertEqual(compat_getenv(test_var), test_str)
+
def test_compat_expanduser(self):
old_home = os.environ.get('HOME')
test_str = 'C:\Documents and Settings\тест\Application Data'
- os.environ['HOME'] = (
- test_str if sys.version_info >= (3, 0)
- else test_str.encode(get_filesystem_encoding()))
+ compat_setenv('HOME', test_str)
self.assertEqual(compat_expanduser('~'), test_str)
- os.environ['HOME'] = old_home
+ compat_setenv('HOME', old_home or '')
def test_all_present(self):
import youtube_dl.compat
self.assertTrue(isinstance(doc.find('chinese').text, compat_str))
self.assertTrue(isinstance(doc.find('foo/bar').text, compat_str))
+ def test_compat_etree_fromstring_doctype(self):
+ xml = '''<?xml version="1.0"?>
+<!DOCTYPE smil PUBLIC "-//W3C//DTD SMIL 2.0//EN" "http://www.w3.org/2001/SMIL20/SMIL20.dtd">
+<smil xmlns="http://www.w3.org/2001/SMIL20/Language"></smil>'''
+ compat_etree_fromstring(xml)
+
+ def test_struct_unpack(self):
+ self.assertEqual(compat_struct_unpack('!B', b'\x00'), (0,))
+
+
if __name__ == '__main__':
unittest.main()
TEST_DIR = os.path.dirname(os.path.abspath(__file__))
+def http_server_port(httpd):
+ if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
+ # In Jython SSLSocket is not a subclass of socket.socket
+ sock = httpd.socket.sock
+ else:
+ sock = httpd.socket
+ return sock.getsockname()[1]
+
+
class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
def log_message(self, format, *args):
pass
self.send_header('Content-Type', 'video/mp4')
self.end_headers()
self.wfile.write(b'\x00\x00\x00\x00\x20\x66\x74[video]')
+ elif self.path == '/302':
+ if sys.version_info[0] == 3:
+ # XXX: Python 3 http server does not allow non-ASCII header values
+ self.send_response(404)
+ self.end_headers()
+ return
+
+ new_url = 'http://localhost:%d/中文.html' % http_server_port(self.server)
+ self.send_response(302)
+ self.send_header(b'Location', new_url.encode('utf-8'))
+ self.end_headers()
+ elif self.path == '/%E4%B8%AD%E6%96%87.html':
+ self.send_response(200)
+ self.send_header('Content-Type', 'text/html; charset=utf-8')
+ self.end_headers()
+ self.wfile.write(b'<html><video src="/vid.mp4" /></html>')
else:
assert False
class TestHTTP(unittest.TestCase):
+ def setUp(self):
+ self.httpd = compat_http_server.HTTPServer(
+ ('localhost', 0), HTTPTestRequestHandler)
+ self.port = http_server_port(self.httpd)
+ self.server_thread = threading.Thread(target=self.httpd.serve_forever)
+ self.server_thread.daemon = True
+ self.server_thread.start()
+
+ def test_unicode_path_redirection(self):
+ # XXX: Python 3 http server does not allow non-ASCII header values
+ if sys.version_info[0] == 3:
+ return
+
+ ydl = YoutubeDL({'logger': FakeLogger()})
+ r = ydl.extract_info('http://localhost:%d/302' % self.port)
+ self.assertEqual(r['url'], 'http://localhost:%d/vid.mp4' % self.port)
+
+
+class TestHTTPS(unittest.TestCase):
def setUp(self):
certfn = os.path.join(TEST_DIR, 'testcert.pem')
self.httpd = compat_http_server.HTTPServer(
('localhost', 0), HTTPTestRequestHandler)
self.httpd.socket = ssl.wrap_socket(
self.httpd.socket, certfile=certfn, server_side=True)
- if os.name == 'java':
- # In Jython SSLSocket is not a subclass of socket.socket
- sock = self.httpd.socket.sock
- else:
- sock = self.httpd.socket
- self.port = sock.getsockname()[1]
+ self.port = http_server_port(self.httpd)
self.server_thread = threading.Thread(target=self.httpd.serve_forever)
self.server_thread.daemon = True
self.server_thread.start()
def setUp(self):
self.proxy = compat_http_server.HTTPServer(
('localhost', 0), _build_proxy_handler('normal'))
- self.port = self.proxy.socket.getsockname()[1]
+ self.port = http_server_port(self.proxy)
self.proxy_thread = threading.Thread(target=self.proxy.serve_forever)
self.proxy_thread.daemon = True
self.proxy_thread.start()
self.cn_proxy = compat_http_server.HTTPServer(
('localhost', 0), _build_proxy_handler('cn'))
- self.cn_port = self.cn_proxy.socket.getsockname()[1]
+ self.cn_port = http_server_port(self.cn_proxy)
self.cn_proxy_thread = threading.Thread(target=self.cn_proxy.serve_forever)
self.cn_proxy_thread.daemon = True
self.cn_proxy_thread.start()
--- /dev/null
+#!/usr/bin/env python
+# coding: utf-8
+from __future__ import unicode_literals
+
+# Allow direct execution
+import os
+import sys
+import unittest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import random
+import subprocess
+
+from test.helper import (
+ FakeYDL,
+ get_params,
+)
+from youtube_dl.compat import (
+ compat_str,
+ compat_urllib_request,
+)
+
+
+class TestMultipleSocks(unittest.TestCase):
+ @staticmethod
+ def _check_params(attrs):
+ params = get_params()
+ for attr in attrs:
+ if attr not in params:
+ print('Missing %s. Skipping.' % attr)
+ return
+ return params
+
+ def test_proxy_http(self):
+ params = self._check_params(['primary_proxy', 'primary_server_ip'])
+ if params is None:
+ return
+ ydl = FakeYDL({
+ 'proxy': params['primary_proxy']
+ })
+ self.assertEqual(
+ ydl.urlopen('http://yt-dl.org/ip').read().decode('utf-8'),
+ params['primary_server_ip'])
+
+ def test_proxy_https(self):
+ params = self._check_params(['primary_proxy', 'primary_server_ip'])
+ if params is None:
+ return
+ ydl = FakeYDL({
+ 'proxy': params['primary_proxy']
+ })
+ self.assertEqual(
+ ydl.urlopen('https://yt-dl.org/ip').read().decode('utf-8'),
+ params['primary_server_ip'])
+
+ def test_secondary_proxy_http(self):
+ params = self._check_params(['secondary_proxy', 'secondary_server_ip'])
+ if params is None:
+ return
+ ydl = FakeYDL()
+ req = compat_urllib_request.Request('http://yt-dl.org/ip')
+ req.add_header('Ytdl-request-proxy', params['secondary_proxy'])
+ self.assertEqual(
+ ydl.urlopen(req).read().decode('utf-8'),
+ params['secondary_server_ip'])
+
+ def test_secondary_proxy_https(self):
+ params = self._check_params(['secondary_proxy', 'secondary_server_ip'])
+ if params is None:
+ return
+ ydl = FakeYDL()
+ req = compat_urllib_request.Request('https://yt-dl.org/ip')
+ req.add_header('Ytdl-request-proxy', params['secondary_proxy'])
+ self.assertEqual(
+ ydl.urlopen(req).read().decode('utf-8'),
+ params['secondary_server_ip'])
+
+
+class TestSocks(unittest.TestCase):
+ _SKIP_SOCKS_TEST = True
+
+ def setUp(self):
+ if self._SKIP_SOCKS_TEST:
+ return
+
+ self.port = random.randint(20000, 30000)
+ self.server_process = subprocess.Popen([
+ 'srelay', '-f', '-i', '127.0.0.1:%d' % self.port],
+ stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+ def tearDown(self):
+ if self._SKIP_SOCKS_TEST:
+ return
+
+ self.server_process.terminate()
+ self.server_process.communicate()
+
+ def _get_ip(self, protocol):
+ if self._SKIP_SOCKS_TEST:
+ return '127.0.0.1'
+
+ ydl = FakeYDL({
+ 'proxy': '%s://127.0.0.1:%d' % (protocol, self.port),
+ })
+ return ydl.urlopen('http://yt-dl.org/ip').read().decode('utf-8')
+
+ def test_socks4(self):
+ self.assertTrue(isinstance(self._get_ip('socks4'), compat_str))
+
+ def test_socks4a(self):
+ self.assertTrue(isinstance(self._get_ip('socks4a'), compat_str))
+
+ def test_socks5(self):
+ self.assertTrue(isinstance(self._get_ip('socks5'), compat_str))
+
+
+if __name__ == '__main__':
+ unittest.main()
sanitize_path,
prepend_extension,
replace_extension,
+ remove_start,
+ remove_end,
remove_quotes,
shell_quote,
smuggle_url,
str_to_int,
strip_jsonp,
- struct_unpack,
timeconvert,
unescapeHTML,
unified_strdate,
self.assertEqual('yes_no', sanitize_filename('yes? no', restricted=True))
self.assertEqual('this_-_that', sanitize_filename('this: that', restricted=True))
- tests = 'a\xe4b\u4e2d\u56fd\u7684c'
- self.assertEqual(sanitize_filename(tests, restricted=True), 'a_b_c')
+ tests = 'aäb\u4e2d\u56fd\u7684c'
+ self.assertEqual(sanitize_filename(tests, restricted=True), 'aab_c')
self.assertTrue(sanitize_filename('\xf6', restricted=True) != '') # No empty filename
forbidden = '"\0\\/&!: \'\t\n()[]{}$;`^,#'
self.assertTrue(sanitize_filename('-', restricted=True) != '')
self.assertTrue(sanitize_filename(':', restricted=True) != '')
+ self.assertEqual(sanitize_filename(
+ 'ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ', restricted=True),
+ 'AAAAAAAECEEEEIIIIDNOOOOOOOOEUUUUUYPssaaaaaaaeceeeeiiiionooooooooeuuuuuypy')
+
def test_sanitize_ids(self):
self.assertEqual(sanitize_filename('_n_cd26wFpw', is_id=True), '_n_cd26wFpw')
self.assertEqual(sanitize_filename('_BD_eEpuzXw', is_id=True), '_BD_eEpuzXw')
self.assertEqual(replace_extension('.abc', 'temp'), '.abc.temp')
self.assertEqual(replace_extension('.abc.ext', 'temp'), '.abc.temp')
+ def test_remove_start(self):
+ self.assertEqual(remove_start(None, 'A - '), None)
+ self.assertEqual(remove_start('A - B', 'A - '), 'B')
+ self.assertEqual(remove_start('B - A', 'A - '), 'B - A')
+
+ def test_remove_end(self):
+ self.assertEqual(remove_end(None, ' - B'), None)
+ self.assertEqual(remove_end('A - B', ' - B'), 'A')
+ self.assertEqual(remove_end('B - A', ' - B'), 'B - A')
+
def test_remove_quotes(self):
self.assertEqual(remove_quotes(None), None)
self.assertEqual(remove_quotes('"'), '"')
testPL(5, 2, (2, 99), [2, 3, 4])
testPL(5, 2, (20, 99), [])
- def test_struct_unpack(self):
- self.assertEqual(struct_unpack('!B', b'\x00'), (0,))
-
def test_read_batch_urls(self):
f = io.StringIO('''\xef\xbb\xbf foo
bar\r
json_code = js_to_json(inp)
self.assertEqual(json.loads(json_code), json.loads(inp))
+ inp = '''{
+ 0:{src:'skipped', type: 'application/dash+xml'},
+ 1:{src:'skipped', type: 'application/vnd.apple.mpegURL'},
+ }'''
+ self.assertEqual(js_to_json(inp), '''{
+ "0":{"src":"skipped", "type": "application/dash+xml"},
+ "1":{"src":"skipped", "type": "application/vnd.apple.mpegURL"}
+ }''')
+
def test_js_to_json_edgecases(self):
on = js_to_json("{abc_def:'1\\'\\\\2\\\\\\'3\"4'}")
self.assertEqual(json.loads(on), {"abc_def": "1'\\2\\'3\"4"})
on = js_to_json('{"abc": "def",}')
self.assertEqual(json.loads(on), {'abc': 'def'})
+ on = js_to_json('{ 0: /* " \n */ ",]" , }')
+ self.assertEqual(json.loads(on), {'0': ',]'})
+
+ on = js_to_json(r'["<p>x<\/p>"]')
+ self.assertEqual(json.loads(on), ['<p>x</p>'])
+
+ on = js_to_json(r'["\xaa"]')
+ self.assertEqual(json.loads(on), ['\u00aa'])
+
+ on = js_to_json("['a\\\nb']")
+ self.assertEqual(json.loads(on), ['ab'])
+
+ on = js_to_json('{0xff:0xff}')
+ self.assertEqual(json.loads(on), {'255': 255})
+
+ on = js_to_json('{077:077}')
+ self.assertEqual(json.loads(on), {'63': 63})
+
+ on = js_to_json('{42:42}')
+ self.assertEqual(json.loads(on), {'42': 42})
+
def test_extract_attributes(self):
self.assertEqual(extract_attributes('<e x="y">'), {'x': 'y'})
self.assertEqual(extract_attributes("<e x='y'>"), {'x': 'y'})
defaultargs = test --exclude test_download.py --exclude test_age_restriction.py
--exclude test_subtitles.py --exclude test_write_annotations.py
--exclude test_youtube_lists.py --exclude test_iqiyi_sdk_interpreter.py
+ --exclude test_socks.py
commands = nosetests --verbose {posargs:{[testenv]defaultargs}} # --with-coverage --cover-package=youtube_dl --cover-html
# test.test_download:TestDownload.test_NowVideo
PostProcessingError,
preferredencoding,
prepend_extension,
+ register_socks_protocols,
render_table,
replace_extension,
SameFileError,
['fribidi', '-c', 'UTF-8'] + width_args, **sp_kwargs)
self._output_channel = os.fdopen(master, 'rb')
except OSError as ose:
- if ose.errno == 2:
+ if ose.errno == errno.ENOENT:
self.report_warning('Could not find fribidi executable, ignoring --bidi-workaround . Make sure that fribidi is an executable file in one of the directories in your $PATH.')
else:
raise
for ph in self.params.get('progress_hooks', []):
self.add_progress_hook(ph)
+ register_socks_protocols()
+
def warn_if_short_id(self, argv):
# short YouTube ID starting with dash?
idxs = [
is_id=(k == 'id'))
template_dict = dict((k, sanitize(k, v))
for k, v in template_dict.items()
- if v is not None)
+ if v is not None and not isinstance(v, (list, tuple, dict)))
template_dict = collections.defaultdict(lambda: 'NA', template_dict)
outtmpl = self.params.get('outtmpl', DEFAULT_OUTTMPL)
result_type = ie_result.get('_type', 'video')
if result_type in ('url', 'url_transparent'):
+ ie_result['url'] = sanitize_url(ie_result['url'])
extract_flat = self.params.get('extract_flat', False)
if ((extract_flat == 'in_playlist' and 'playlist' in extra_info) or
extract_flat is True):
# Just a single file
success = dl(filename, info_dict)
except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
- self.report_error('unable to download video data: %s' % str(err))
+ self.report_error('unable to download video data: %s' % error_to_compat_str(err))
return
except (OSError, IOError) as err:
raise UnavailableVideoError(err)
if opts_cookiefile is None:
self.cookiejar = compat_cookiejar.CookieJar()
else:
+ opts_cookiefile = compat_expanduser(opts_cookiefile)
self.cookiejar = compat_cookiejar.MozillaCookieJar(
opts_cookiefile)
if os.access(opts_cookiefile, os.R_OK):
from .compat import (
compat_expanduser,
compat_getpass,
- compat_print,
compat_shlex_split,
workaround_optparse_bug9161,
)
# Custom HTTP headers
if opts.headers is not None:
for h in opts.headers:
- if h.find(':', 1) < 0:
+ if ':' not in h:
parser.error('wrong header formatting, it should be key:value, not "%s"' % h)
- key, value = h.split(':', 2)
+ key, value = h.split(':', 1)
if opts.verbose:
write_string('[debug] Adding header from command line option %s:%s\n' % (key, value))
std_headers[key] = value
# Dump user agent
if opts.dump_user_agent:
- compat_print(std_headers['User-Agent'])
+ write_string(std_headers['User-Agent'] + '\n', out=sys.stdout)
sys.exit(0)
# Batch file verification
if opts.batchfile == '-':
batchfd = sys.stdin
else:
- batchfd = io.open(opts.batchfile, 'r', encoding='utf-8', errors='ignore')
+ batchfd = io.open(
+ compat_expanduser(opts.batchfile),
+ 'r', encoding='utf-8', errors='ignore')
batch_urls = read_batch_urls(batchfd)
if opts.verbose:
write_string('[debug] Batch file urls: ' + repr(batch_urls) + '\n')
if opts.list_extractors:
for ie in list_extractors(opts.age_limit):
- compat_print(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else ''))
+ write_string(ie.IE_NAME + (' (CURRENTLY BROKEN)' if not ie._WORKING else '') + '\n', out=sys.stdout)
matchedUrls = [url for url in all_urls if ie.suitable(url)]
for mu in matchedUrls:
- compat_print(' ' + mu)
+ write_string(' ' + mu + '\n', out=sys.stdout)
sys.exit(0)
if opts.list_extractor_descriptions:
for ie in list_extractors(opts.age_limit):
_SEARCHES = ('cute kittens', 'slithering pythons', 'falling cat', 'angry poodle', 'purple fish', 'running tortoise', 'sleeping bunny', 'burping cow')
_COUNTS = ('', '5', '10', 'all')
desc += ' (Example: "%s%s:%s" )' % (ie.SEARCH_KEY, random.choice(_COUNTS), random.choice(_SEARCHES))
- compat_print(desc)
+ write_string(desc + '\n', out=sys.stdout)
sys.exit(0)
# Conflicting, missing and erroneous options
try:
if opts.load_info_filename is not None:
- retcode = ydl.download_with_info_file(opts.load_info_filename)
+ retcode = ydl.download_with_info_file(compat_expanduser(opts.load_info_filename))
else:
retcode = ydl.download(all_urls)
except MaxDownloadsReached:
import shlex
import shutil
import socket
+import struct
import subprocess
import sys
import itertools
except ImportError: # Python 2.6
from xml.parsers.expat import ExpatError as compat_xml_parse_error
+
+etree = xml.etree.ElementTree
+
+
+class _TreeBuilder(etree.TreeBuilder):
+ def doctype(self, name, pubid, system):
+ pass
+
if sys.version_info[0] >= 3:
- compat_etree_fromstring = xml.etree.ElementTree.fromstring
+ def compat_etree_fromstring(text):
+ return etree.XML(text, parser=etree.XMLParser(target=_TreeBuilder()))
else:
# python 2.x tries to encode unicode strings with ascii (see the
# XMLParser._fixtext method)
- etree = xml.etree.ElementTree
-
try:
_etree_iter = etree.Element.iter
except AttributeError: # Python <=2.6
# 2.7 source
def _XML(text, parser=None):
if not parser:
- parser = etree.XMLParser(target=etree.TreeBuilder())
+ parser = etree.XMLParser(target=_TreeBuilder())
parser.feed(text)
return parser.close()
return el
def compat_etree_fromstring(text):
- doc = _XML(text, parser=etree.XMLParser(target=etree.TreeBuilder(element_factory=_element_factory)))
+ doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
for el in _etree_iter(doc):
if el.text is not None and isinstance(el.text, bytes):
el.text = el.text.decode('utf-8')
return parsed_result
try:
- from shlex import quote as shlex_quote
+ from shlex import quote as compat_shlex_quote
except ImportError: # Python < 3.3
- def shlex_quote(s):
+ def compat_shlex_quote(s):
if re.match(r'^[-_\w./]+$', s):
return s
else:
if sys.version_info >= (3, 0):
compat_getenv = os.getenv
compat_expanduser = os.path.expanduser
+
+ def compat_setenv(key, value, env=os.environ):
+ env[key] = value
else:
# Environment variables should be decoded with filesystem encoding.
# Otherwise it will fail if any non-ASCII characters present (see #3854 #3217 #2918)
env = env.decode(get_filesystem_encoding())
return env
+ def compat_setenv(key, value, env=os.environ):
+ def encode(v):
+ from .utils import get_filesystem_encoding
+ return v.encode(get_filesystem_encoding()) if isinstance(v, compat_str) else v
+ env[encode(key)] = encode(value)
+
# HACK: The default implementations of os.path.expanduser from cpython do not decode
# environment variables with filesystem encoding. We will work around this by
# providing adjusted implementations.
print(s)
-try:
- subprocess_check_output = subprocess.check_output
-except AttributeError:
- def subprocess_check_output(*args, **kwargs):
- assert 'input' not in kwargs
- p = subprocess.Popen(*args, stdout=subprocess.PIPE, **kwargs)
- output, _ = p.communicate()
- ret = p.poll()
- if ret:
- raise subprocess.CalledProcessError(ret, p.args, output=output)
- return output
-
if sys.version_info < (3, 0) and sys.platform == 'win32':
def compat_getpass(prompt, *args, **kwargs):
if isinstance(prompt, compat_str):
else:
compat_getpass = getpass.getpass
+try:
+ compat_input = raw_input
+except NameError: # Python 3
+ compat_input = input
+
# Python < 2.6.5 require kwargs to be bytes
try:
def _testfunc(x):
else:
from tokenize import generate_tokens as compat_tokenize_tokenize
+
+try:
+ struct.pack('!I', 0)
+except TypeError:
+ # In Python 2.6 and 2.7.x < 2.7.7, struct requires a bytes argument
+ # See https://bugs.python.org/issue19099
+ def compat_struct_pack(spec, *args):
+ if isinstance(spec, compat_str):
+ spec = spec.encode('ascii')
+ return struct.pack(spec, *args)
+
+ def compat_struct_unpack(spec, *args):
+ if isinstance(spec, compat_str):
+ spec = spec.encode('ascii')
+ return struct.unpack(spec, *args)
+else:
+ compat_struct_pack = struct.pack
+ compat_struct_unpack = struct.unpack
+
+
__all__ = [
'compat_HTMLParser',
'compat_HTTPError',
'compat_html_entities',
'compat_http_client',
'compat_http_server',
+ 'compat_input',
'compat_itertools_count',
'compat_kwargs',
'compat_ord',
'compat_os_name',
'compat_parse_qs',
'compat_print',
+ 'compat_setenv',
+ 'compat_shlex_quote',
'compat_shlex_split',
'compat_socket_create_connection',
'compat_str',
+ 'compat_struct_pack',
+ 'compat_struct_unpack',
'compat_subprocess_get_DEVNULL',
'compat_tokenize_tokenize',
'compat_urllib_error',
'compat_urlretrieve',
'compat_xml_parse_error',
'compat_xpath',
- 'shlex_quote',
- 'subprocess_check_output',
'workaround_optparse_bug9161',
]
import re
from .common import FileDownloader
+from ..compat import compat_setenv
from ..postprocessor.ffmpeg import FFmpegPostProcessor, EXT_TO_OUT_FORMATS
from ..utils import (
cli_option,
'-headers',
''.join('%s: %s\r\n' % (key, val) for key, val in headers.items())]
+ env = None
+ proxy = self.params.get('proxy')
+ if proxy:
+ if not re.match(r'^[\da-zA-Z]+://', proxy):
+ proxy = 'http://%s' % proxy
+ # Since December 2015 ffmpeg supports -http_proxy option (see
+ # http://git.videolan.org/?p=ffmpeg.git;a=commit;h=b4eb1f29ebddd60c41a2eb39f5af701e38e0d3fd)
+ # We could switch to the following code if we are able to detect version properly
+ # args += ['-http_proxy', proxy]
+ env = os.environ.copy()
+ compat_setenv('HTTP_PROXY', proxy, env=env)
+
protocol = info_dict.get('protocol')
if protocol == 'rtmp':
args += ['-rtmp_live', 'live']
args += ['-i', url, '-c', 'copy']
- if protocol == 'm3u8':
+ if protocol in ('m3u8', 'm3u8_native'):
if self.params.get('hls_use_mpegts', False) or tmpfilename == '-':
args += ['-f', 'mpegts']
else:
self._debug_cmd(args)
- proc = subprocess.Popen(args, stdin=subprocess.PIPE)
+ proc = subprocess.Popen(args, stdin=subprocess.PIPE, env=env)
try:
retval = proc.wait()
except KeyboardInterrupt:
compat_urlparse,
compat_urllib_error,
compat_urllib_parse_urlparse,
+ compat_struct_pack,
+ compat_struct_unpack,
)
from ..utils import (
encodeFilename,
fix_xml_ampersands,
sanitize_open,
- struct_pack,
- struct_unpack,
xpath_text,
)
+class DataTruncatedError(Exception):
+ pass
+
+
class FlvReader(io.BytesIO):
"""
Reader for Flv files
The file format is documented in https://www.adobe.com/devnet/f4v.html
"""
+ def read_bytes(self, n):
+ data = self.read(n)
+ if len(data) < n:
+ raise DataTruncatedError(
+ 'FlvReader error: need %d bytes while only %d bytes got' % (
+ n, len(data)))
+ return data
+
# Utility functions for reading numbers and strings
def read_unsigned_long_long(self):
- return struct_unpack('!Q', self.read(8))[0]
+ return compat_struct_unpack('!Q', self.read_bytes(8))[0]
def read_unsigned_int(self):
- return struct_unpack('!I', self.read(4))[0]
+ return compat_struct_unpack('!I', self.read_bytes(4))[0]
def read_unsigned_char(self):
- return struct_unpack('!B', self.read(1))[0]
+ return compat_struct_unpack('!B', self.read_bytes(1))[0]
def read_string(self):
res = b''
while True:
- char = self.read(1)
+ char = self.read_bytes(1)
if char == b'\x00':
break
res += char
Read a box and return the info as a tuple: (box_size, box_type, box_data)
"""
real_size = size = self.read_unsigned_int()
- box_type = self.read(4)
+ box_type = self.read_bytes(4)
header_end = 8
if size == 1:
real_size = self.read_unsigned_long_long()
header_end = 16
- return real_size, box_type, self.read(real_size - header_end)
+ return real_size, box_type, self.read_bytes(real_size - header_end)
def read_asrt(self):
# version
self.read_unsigned_char()
# flags
- self.read(3)
+ self.read_bytes(3)
quality_entry_count = self.read_unsigned_char()
# QualityEntryCount
for i in range(quality_entry_count):
# version
self.read_unsigned_char()
# flags
- self.read(3)
+ self.read_bytes(3)
# time scale
self.read_unsigned_int()
# version
self.read_unsigned_char()
# flags
- self.read(3)
+ self.read_bytes(3)
self.read_unsigned_int() # BootstrapinfoVersion
# Profile,Live,Update,Reserved
def write_unsigned_int(stream, val):
- stream.write(struct_pack('!I', val))
+ stream.write(compat_struct_pack('!I', val))
def write_unsigned_int_24(stream, val):
- stream.write(struct_pack('!I', val)[1:])
+ stream.write(compat_struct_pack('!I', val)[1:])
def write_flv_header(stream):
doc = compat_etree_fromstring(manifest)
formats = [(int(f.attrib.get('bitrate', -1)), f)
for f in self._get_unencrypted_media(doc)]
- if requested_bitrate is None:
+ if requested_bitrate is None or len(formats) == 1:
# get the best format
formats = sorted(formats, key=lambda f: f[0])
rate, media = formats[-1]
down.close()
reader = FlvReader(down_data)
while True:
- _, box_type, box_data = reader.read_box_info()
+ try:
+ _, box_type, box_data = reader.read_box_info()
+ except DataTruncatedError:
+ if test:
+ # In tests, segments may be truncated, and thus
+ # FlvReader may not be able to parse the whole
+ # chunk. If so, write the segment as is
+ # See https://github.com/rg3/youtube-dl/issues/9214
+ dest_stream.write(down_data)
+ break
+ raise
if box_type == b'mdat':
dest_stream.write(box_data)
break
import re
from .fragment import FragmentFD
+from .external import FFmpegFD
from ..compat import compat_urlparse
from ..utils import (
FD_NAME = 'hlsnative'
+ @staticmethod
+ def can_download(manifest):
+ UNSUPPORTED_FEATURES = (
+ r'#EXT-X-KEY:METHOD=(?!NONE)', # encrypted streams [1]
+ r'#EXT-X-BYTERANGE', # playlists composed of byte ranges of media files [2]
+
+ # Live streams heuristic does not always work (e.g. geo restricted to Germany
+ # http://hls-geo.daserste.de/i/videoportal/Film/c_620000/622873/format,716451,716457,716450,716458,716459,.mp4.csmil/index_4_av.m3u8?null=0)
+ # r'#EXT-X-MEDIA-SEQUENCE:(?!0$)', # live streams [3]
+
+ # This heuristic also is not correct since segments may not be appended as well.
+ # Twitch vods of finished streams have EXT-X-PLAYLIST-TYPE:EVENT despite
+ # no segments will definitely be appended to the end of the playlist.
+ # r'#EXT-X-PLAYLIST-TYPE:EVENT', # media segments may be appended to the end of
+ # # event media playlists [4]
+
+ # 1. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.4
+ # 2. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.2
+ # 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
+ # 4. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.5
+ )
+ return all(not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES)
+
def real_download(self, filename, info_dict):
man_url = info_dict['url']
self.to_screen('[%s] Downloading m3u8 manifest' % self.FD_NAME)
manifest = self.ydl.urlopen(man_url).read()
s = manifest.decode('utf-8', 'ignore')
+
+ if not self.can_download(s):
+ self.report_warning(
+ 'hlsnative has detected features it does not support, '
+ 'extraction will be delegated to ffmpeg')
+ fd = FFmpegFD(self.ydl, self.params)
+ for ph in self._progress_hooks:
+ fd.add_progress_hook(ph)
+ return fd.real_download(filename, info_dict)
+
fragment_urls = []
for line in s.splitlines():
line = line.strip()
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import calendar
+import re
+import time
+
+from .amp import AMPIE
+from .common import InfoExtractor
+from ..compat import compat_urlparse
+
+
+class AbcNewsVideoIE(AMPIE):
+ IE_NAME = 'abcnews:video'
+ _VALID_URL = 'http://abcnews.go.com/[^/]+/video/(?P<display_id>[0-9a-z-]+)-(?P<id>\d+)'
+
+ _TESTS = [{
+ 'url': 'http://abcnews.go.com/ThisWeek/video/week-exclusive-irans-foreign-minister-zarif-20411932',
+ 'info_dict': {
+ 'id': '20411932',
+ 'ext': 'mp4',
+ 'display_id': 'week-exclusive-irans-foreign-minister-zarif',
+ 'title': '\'This Week\' Exclusive: Iran\'s Foreign Minister Zarif',
+ 'description': 'George Stephanopoulos goes one-on-one with Iranian Foreign Minister Dr. Javad Zarif.',
+ 'duration': 180,
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
+ }, {
+ 'url': 'http://abcnews.go.com/2020/video/2020-husband-stands-teacher-jail-student-affairs-26119478',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ display_id = mobj.group('display_id')
+ video_id = mobj.group('id')
+ info_dict = self._extract_feed_info(
+ 'http://abcnews.go.com/video/itemfeed?id=%s' % video_id)
+ info_dict.update({
+ 'id': video_id,
+ 'display_id': display_id,
+ })
+ return info_dict
+
+
+class AbcNewsIE(InfoExtractor):
+ IE_NAME = 'abcnews'
+ _VALID_URL = 'https?://abcnews\.go\.com/(?:[^/]+/)+(?P<display_id>[0-9a-z-]+)/story\?id=(?P<id>\d+)'
+
+ _TESTS = [{
+ 'url': 'http://abcnews.go.com/Blotter/News/dramatic-video-rare-death-job-america/story?id=10498713#.UIhwosWHLjY',
+ 'info_dict': {
+ 'id': '10498713',
+ 'ext': 'flv',
+ 'display_id': 'dramatic-video-rare-death-job-america',
+ 'title': 'Occupational Hazards',
+ 'description': 'Nightline investigates the dangers that lurk at various jobs.',
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ 'upload_date': '20100428',
+ 'timestamp': 1272412800,
+ },
+ 'add_ie': ['AbcNewsVideo'],
+ }, {
+ 'url': 'http://abcnews.go.com/Entertainment/justin-timberlake-performs-stop-feeling-eurovision-2016/story?id=39125818',
+ 'info_dict': {
+ 'id': '39125818',
+ 'ext': 'mp4',
+ 'display_id': 'justin-timberlake-performs-stop-feeling-eurovision-2016',
+ 'title': 'Justin Timberlake Drops Hints For Secret Single',
+ 'description': 'Lara Spencer reports the buzziest stories of the day in "GMA" Pop News.',
+ 'upload_date': '20160515',
+ 'timestamp': 1463329500,
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ # The embedded YouTube video is blocked due to copyright issues
+ 'playlist_items': '1',
+ },
+ 'add_ie': ['AbcNewsVideo'],
+ }, {
+ 'url': 'http://abcnews.go.com/Technology/exclusive-apple-ceo-tim-cook-iphone-cracking-software/story?id=37173343',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ display_id = mobj.group('display_id')
+ video_id = mobj.group('id')
+
+ webpage = self._download_webpage(url, video_id)
+ video_url = self._search_regex(
+ r'window\.abcnvideo\.url\s*=\s*"([^"]+)"', webpage, 'video URL')
+ full_video_url = compat_urlparse.urljoin(url, video_url)
+
+ youtube_url = self._html_search_regex(
+ r'<iframe[^>]+src="(https://www\.youtube\.com/embed/[^"]+)"',
+ webpage, 'YouTube URL', default=None)
+
+ timestamp = None
+ date_str = self._html_search_regex(
+ r'<span[^>]+class="timestamp">([^<]+)</span>',
+ webpage, 'timestamp', fatal=False)
+ if date_str:
+ tz_offset = 0
+ if date_str.endswith(' ET'): # Eastern Time
+ tz_offset = -5
+ date_str = date_str[:-3]
+ date_formats = ['%b. %d, %Y', '%b %d, %Y, %I:%M %p']
+ for date_format in date_formats:
+ try:
+ timestamp = calendar.timegm(time.strptime(date_str.strip(), date_format))
+ except ValueError:
+ continue
+ if timestamp is not None:
+ timestamp -= tz_offset * 3600
+
+ entry = {
+ '_type': 'url_transparent',
+ 'ie_key': AbcNewsVideoIE.ie_key(),
+ 'url': full_video_url,
+ 'id': video_id,
+ 'display_id': display_id,
+ 'timestamp': timestamp,
+ }
+
+ if youtube_url:
+ entries = [entry, self.url_result(youtube_url, 'Youtube')]
+ return self.playlist_result(entries)
+
+ return entry
for media_data in media_content:
media = media_data['@attributes']
media_type = media['type']
- if media_type == 'video/f4m':
+ if media_type in ('video/f4m', 'application/f4m+xml'):
formats.extend(self._extract_f4m_formats(
media['url'] + '?hdcore=3.4.0&plugin=aasp-3.4.0.132.124',
video_id, f4m_id='hds', fatal=False))
media['url'], video_id, 'mp4', m3u8_id='hls', fatal=False))
else:
formats.append({
- 'format_id': media_data['media-category']['@attributes']['label'],
+ 'format_id': media_data.get('media-category', {}).get('@attributes', {}).get('label'),
'url': media['url'],
'tbr': int_or_none(media.get('bitrate')),
'filesize': int_or_none(media.get('fileSize')),
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import base64
+import hashlib
+import json
+import random
+import time
+
+from .common import InfoExtractor
+from ..aes import aes_encrypt
+from ..compat import compat_str
+from ..utils import (
+ bytes_to_intlist,
+ determine_ext,
+ intlist_to_bytes,
+ int_or_none,
+ strip_jsonp,
+)
+
+
+def md5_text(s):
+ if not isinstance(s, compat_str):
+ s = compat_str(s)
+ return hashlib.md5(s.encode('utf-8')).hexdigest()
+
+
+class AnvatoIE(InfoExtractor):
+ # Copied from anvplayer.min.js
+ _ANVACK_TABLE = {
+ 'nbcu_nbcd_desktop_web_prod_93d8ead38ce2024f8f544b78306fbd15895ae5e6': 'NNemUkySjxLyPTKvZRiGntBIjEyK8uqicjMakIaQ',
+ 'nbcu_nbcd_desktop_web_qa_1a6f01bdd0dc45a439043b694c8a031d': 'eSxJUbA2UUKBTXryyQ2d6NuM8oEqaPySvaPzfKNA',
+ 'nbcu_nbcd_desktop_web_acc_eb2ff240a5d4ae9a63d4c297c32716b6c523a129': '89JR3RtUGbvKuuJIiKOMK0SoarLb5MUx8v89RcbP',
+ 'nbcu_nbcd_watchvod_web_prod_e61107507180976724ec8e8319fe24ba5b4b60e1': 'Uc7dFt7MJ9GsBWB5T7iPvLaMSOt8BBxv4hAXk5vv',
+ 'nbcu_nbcd_watchvod_web_qa_42afedba88a36203db5a4c09a5ba29d045302232': 'T12oDYVFP2IaFvxkmYMy5dKxswpLHtGZa4ZAXEi7',
+ 'nbcu_nbcd_watchvod_web_acc_9193214448e2e636b0ffb78abacfd9c4f937c6ca': 'MmobcxUxMedUpohNWwXaOnMjlbiyTOBLL6d46ZpR',
+ 'nbcu_local_monitor_web_acc_f998ad54eaf26acd8ee033eb36f39a7b791c6335': 'QvfIoPYrwsjUCcASiw3AIkVtQob2LtJHfidp9iWg',
+ 'nbcu_cable_monitor_web_acc_a413759603e8bedfcd3c61b14767796e17834077': 'uwVPJLShvJWSs6sWEIuVem7MTF8A4IknMMzIlFto',
+ 'nbcu_nbcd_mcpstage_web_qa_4c43a8f6e95a88dbb40276c0630ba9f693a63a4e': 'PxVYZVwjhgd5TeoPRxL3whssb5OUPnM3zyAzq8GY',
+ 'nbcu_comcast_comcast_web_prod_074080762ad4ce956b26b43fb22abf153443a8c4': 'afnaRZfDyg1Z3WZHdupKfy6xrbAG2MHqe3VfuSwh',
+ 'nbcu_comcast_comcast_web_qa_706103bb93ead3ef70b1de12a0e95e3c4481ade0': 'DcjsVbX9b3uoPlhdriIiovgFQZVxpISZwz0cx1ZK',
+ 'nbcu_comcast_comcastcable_web_prod_669f04817536743563d7331c9293e59fbdbe3d07': '0RwMN2cWy10qhAhOscq3eK7aEe0wqnKt3vJ0WS4D',
+ 'nbcu_comcast_comcastcable_web_qa_3d9d2d66219094127f0f6b09cc3c7bb076e3e1ca': '2r8G9DEya7PCqBceKZgrn2XkXgASjwLMuaFE1Aad',
+ 'hearst_hearst_demo_web_stage_960726dfef3337059a01a78816e43b29ec04dfc7': 'cuZBPXTR6kSdoTCVXwk5KGA8rk3NrgGn4H6e9Dsp',
+ 'anvato_mcpqa_demo_web_stage_18b55e00db5a13faa8d03ae6e41f6f5bcb15b922': 'IOaaLQ8ymqVyem14QuAvE5SndQynTcH5CrLkU2Ih',
+ 'anvato_nextmedia_demo_web_stage_9787d56a02ff6b9f43e9a2b0920d8ca88beb5818': 'Pqu9zVzI1ApiIzbVA3VkGBEQHvdKSUuKpD6s2uaR',
+ 'anvato_scripps_app_web_prod_0837996dbe373629133857ae9eb72e740424d80a': 'du1ccmn7RxzgizwbWU7hyUaGodNlJn7HtXI0WgXW',
+ 'anvato_scripps_app_web_stage_360797e00fe2826be142155c4618cc52fce6c26c': '2PMrQ0BRoqCWl7nzphj0GouIMEh2mZYivAT0S1Su',
+ 'fs2go_fs2go_go_all_prod_21934911ccfafc03a075894ead2260d11e2ddd24': 'RcuHlKikW2IJw6HvVoEkqq2UsuEJlbEl11pWXs4Q',
+ 'fs2go_fs2go_go_web_prod_ead4b0eec7460c1a07783808db21b49cf1f2f9a7': '4K0HTT2u1zkQA2MaGaZmkLa1BthGSBdr7jllrhk5',
+ 'fs2go_fs2go_go_web_stage_407585454a4400355d4391691c67f361': 'ftnc37VKRJBmHfoGGi3kT05bHyeJzilEzhKJCyl3',
+ 'fs2go_fs2go_go_android_stage_44b714db6f8477f29afcba15a41e1d30': 'CtxpPvVpo6AbZGomYUhkKs7juHZwNml9b9J0J2gI',
+ 'anvato_cbslocal_app_web_prod_547f3e49241ef0e5d30c79b2efbca5d92c698f67': 'Pw0XX5KBDsyRnPS0R2JrSrXftsy8Jnz5pAjaYC8s',
+ 'anvato_cbslocal_app_web_stage_547a5f096594cd3e00620c6f825cad1096d28c80': '37OBUhX2uwNyKhhrNzSSNHSRPZpApC3trdqDBpuz',
+ 'fs2go_att_att_web_prod_1042dddd089a05438b6a08f972941176f699ffd8': 'JLcF20JwYvpv6uAGcLWIaV12jKwaL1R8us4b6Zkg',
+ 'fs2go_att_att_web_stage_807c5001955fc114a3331fe027ddc76e': 'gbu1oO1y0JiOFh4SUipt86P288JHpyjSqolrrT1x',
+ 'fs2go_fs2go_tudor_web_prod_a7dd8e5a7cdc830cae55eae6f3e9fee5ee49eb9b': 'ipcp87VCEZXPPe868j3orLqzc03oTy7DXsGkAXXH',
+ 'anvato_mhz_app_web_prod_b808218b30de7fdf60340cbd9831512bc1bf6d37': 'Stlm5Gs6BEhJLRTZHcNquyzxGqr23EuFmE5DCgjX',
+ 'fs2go_charter_charter_web_stage_c2c6e5a68375a1bf00fff213d3ff8f61a835a54c': 'Lz4hbJp1fwL6jlcz4M2PMzghM4jp4aAmybtT5dPc',
+ 'fs2go_charter_charter_web_prod_ebfe3b10f1af215a7321cd3d629e0b81dfa6fa8c': 'vUJsK345A1bVmyYDRhZX0lqFIgVXuqhmuyp1EtPK',
+ 'anvato_epfox_app_web_prod_b3373168e12f423f41504f207000188daf88251b': 'GDKq1ixvX3MoBNdU5IOYmYa2DTUXYOozPjrCJnW7',
+ 'anvato_epfox_app_web_stage_a3c2ce60f8f83ef374a88b68ee73a950f8ab87ce': '2jz2NH4BsXMaDsoJ5qkHMbcczAfIReo2eFYuVC1C',
+ 'fs2go_verizon_verizon_web_stage_08e6df0354a4803f1b1f2428b5a9a382e8dbcd62': 'rKTVapNaAcmnUbGL4ZcuOoY4SE7VmZSQsblPFr7e',
+ 'fs2go_verizon_verizon_web_prod_f909564cb606eff1f731b5e22e0928676732c445': 'qLSUuHerM3u9eNPzaHyUK52obai5MvE4XDJfqYe1',
+ 'fs2go_foxcom_synd_web_stage_f7b9091f00ea25a4fdaaae77fca5b54cdc7e7043': '96VKF2vLd24fFiDfwPFpzM5llFN4TiIGAlodE0Re',
+ 'fs2go_foxcom_synd_web_prod_0f2cdd64d87e4ab6a1d54aada0ff7a7c8387a064': 'agiPjbXEyEZUkbuhcnmVPhe9NNVbDjCFq2xkcx51',
+ 'anvato_own_app_web_stage_1214ade5d28422c4dae9d03c1243aba0563c4dba': 'mzhamNac3swG4WsJAiUTacnGIODi6SWeVWk5D7ho',
+ 'anvato_own_app_web_prod_944e162ed927ec3e9ed13eb68ed2f1008ee7565e': '9TSxh6G2TXOLBoYm9ro3LdNjjvnXpKb8UR8KoIP9',
+ 'anvato_scripps_app_ftv_prod_a10a10468edd5afb16fb48171c03b956176afad1': 'COJ2i2UIPK7xZqIWswxe7FaVBOVgRkP1F6O6qGoH',
+ 'anvato_scripps_app_ftv_stage_77d3ad2bdb021ec37ca2e35eb09acd396a974c9a': 'Q7nnopNLe2PPfGLOTYBqxSaRpl209IhqaEuDZi1F',
+ 'anvato_univision_app_web_stage_551236ef07a0e17718c3995c35586b5ed8cb5031': 'D92PoLS6UitwxDRA191HUGT9OYcOjV6mPMa5wNyo',
+ 'anvato_univision_app_web_prod_039a5c0a6009e637ae8ac906718a79911e0e65e1': '5mVS5u4SQjtw6NGw2uhMbKEIONIiLqRKck5RwQLR',
+ 'nbcu_cnbc_springfield_ios_prod_670207fae43d6e9a94c351688851a2ce': 'M7fqCCIP9lW53oJbHs19OlJlpDrVyc2OL8gNeuTa',
+ 'nbcu_cnbc_springfieldvod_ios_prod_7a5f04b1ceceb0e9c9e2264a44aa236e08e034c2': 'Yia6QbJahW0S7K1I0drksimhZb4UFq92xLBmmMvk',
+ 'anvato_cox_app_web_prod_ce45cda237969f93e7130f50ee8bb6280c1484ab': 'cc0miZexpFtdoqZGvdhfXsLy7FXjRAOgb9V0f5fZ',
+ 'anvato_cox_app_web_stage_c23dbe016a8e9d8c7101d10172b92434f6088bf9': 'yivU3MYHd2eDZcOfmLbINVtqxyecKTOp8OjOuoGJ',
+ 'anvato_chnzero_app_web_stage_b1164d1352b579e792e542fddf13ee34c0eeb46b': 'A76QkXMmVH8lTCfU15xva1mZnSVcqeY4Xb22Kp7m',
+ 'anvato_chnzero_app_web_prod_253d358928dc08ec161eda2389d53707288a730c': 'OA5QI3ZWZZkdtUEDqh28AH8GedsF6FqzJI32596b',
+ 'anvato_discovery_vodpoc_web_stage_9fa7077b5e8af1f8355f65d4fb8d2e0e9d54e2b7': 'q3oT191tTQ5g3JCP67PkjLASI9s16DuWZ6fYmry3',
+ 'anvato_discovery_vodpoc_web_prod_688614983167a1af6cdf6d76343fda10a65223c1': 'qRvRQCTVHd0VVOHsMvvfidyWmlYVrTbjby7WqIuK',
+ 'nbcu_cnbc_springfieldvod_ftv_stage_826040aad1925a46ac5dfb4b3c5143e648c6a30d': 'JQaSb5a8Tz0PT4ti329DNmzDO30TnngTHmvX8Vua',
+ 'nbcu_cnbc_springfield_ftv_stage_826040aad1925a46ac5dfb4b3c5143e648c6a30d': 'JQaSb5a8Tz0PT4ti329DNmzDO30TnngTHmvX8Vua',
+ 'nbcu_nbcd_capture_web_stage_4dd9d585bfb984ebf856dee35db027b2465cc4ae': '0j1Ov4Vopyi2HpBZJYdL2m8ERJVGYh3nNpzPiO8F',
+ 'nbcu_nbcd_watch3_android_prod_7712ca5fcf1c22f19ec1870a9650f9c37db22dcf': '3LN2UB3rPUAMu7ZriWkHky9vpLMXYha8JbSnxBlx',
+ 'nbcu_nbcd_watchvod3_android_prod_0910a3a4692d57c0b5ff4316075bc5d096be45b9': 'mJagcQ2II30vUOAauOXne7ERwbf5S9nlB3IP17lQ',
+ 'anvato_scripps_app_atv_prod_790deda22e16e71e83df58f880cd389908a45d52': 'CB6trI1mpoDIM5o54DNTsji90NDBQPZ4z4RqBNSH',
+ 'nbcu_nbcd_watchv4_android_prod_ff67cef9cb409158c6f8c3533edddadd0b750507': 'j8CHQCUWjlYERj4NFRmUYOND85QNbHViH09UwuKm',
+ 'nbcu_nbcd_watchvodv4_android_prod_a814d781609989dea6a629d50ae4c7ad8cc8e907': 'rkVnUXxdA9rawVLUlDQtMue9Y4Q7lFEaIotcUhjt',
+ 'rvVKpA50qlOPLFxMjrCGf5pdkdQDm7qn': '1J7ZkY5Qz5lMLi93QOH9IveE7EYB3rLl',
+ 'nbcu_dtv_local_web_prod_b266cf49defe255fd4426a97e27c09e513e9f82f': 'HuLnJDqzLa4saCzYMJ79zDRSQpEduw1TzjMNQu2b',
+ 'nbcu_att_local_web_prod_4cef038b2d969a6b7d700a56a599040b6a619f67': 'Q0Em5VDc2KpydUrVwzWRXAwoNBulWUxCq2faK0AV',
+ 'nbcu_dish_local_web_prod_c56dcaf2da2e9157a4266c82a78195f1dd570f6b': 'bC1LWmRz9ayj2AlzizeJ1HuhTfIaJGsDBnZNgoRg',
+ 'nbcu_verizon_local_web_prod_88bebd2ce006d4ed980de8133496f9a74cb9b3e1': 'wzhDKJZpgvUSS1EQvpCQP8Q59qVzcPixqDGJefSk',
+ 'nbcu_charter_local_web_prod_9ad90f7fc4023643bb718f0fe0fd5beea2382a50': 'PyNbxNhEWLzy1ZvWEQelRuIQY88Eub7xbSVRMdfT',
+ 'nbcu_suddenlink_local_web_prod_20fb711725cac224baa1c1cb0b1c324d25e97178': '0Rph41lPXZbb3fqeXtHjjbxfSrNbtZp1Ygq7Jypa',
+ 'nbcu_wow_local_web_prod_652d9ce4f552d9c2e7b5b1ed37b8cb48155174ad': 'qayIBZ70w1dItm2zS42AptXnxW15mkjRrwnBjMPv',
+ 'nbcu_centurylink_local_web_prod_2034402b029bf3e837ad46814d9e4b1d1345ccd5': 'StePcPMkjsX51PcizLdLRMzxMEl5k2FlsMLUNV4k',
+ 'nbcu_atlanticbrd_local_web_prod_8d5f5ecbf7f7b2f5e6d908dd75d90ae3565f682e': 'NtYLb4TFUS0pRs3XTkyO5sbVGYjVf17bVbjaGscI',
+ 'nbcu_nbcd_watchvod_web_dev_08bc05699be47c4f31d5080263a8cfadc16d0f7c': 'hwxi2dgDoSWgfmVVXOYZm14uuvku4QfopstXckhr',
+ 'anvato_nextmedia_app_web_prod_a4fa8c7204aa65e71044b57aaf63711980cfe5a0': 'tQN1oGPYY1nM85rJYePWGcIb92TG0gSqoVpQTWOw',
+ 'anvato_mcp_lin_web_prod_4c36fbfd4d8d8ecae6488656e21ac6d1ac972749': 'GUXNf5ZDX2jFUpu4WT2Go4DJ5nhUCzpnwDRRUx1K',
+ 'anvato_mcp_univision_web_prod_37fe34850c99a3b5cdb71dab10a417dd5cdecafa': 'bLDYF8JqfG42b7bwKEgQiU9E2LTIAtnKzSgYpFUH',
+ 'anvato_mcp_fs2go_web_prod_c7b90a93e171469cdca00a931211a2f556370d0a': 'icgGoYGipQMMSEvhplZX1pwbN69srwKYWksz3xWK',
+ 'anvato_mcp_sps_web_prod_54bdc90dd6ba21710e9f7074338365bba28da336': 'fA2iQdI7RDpynqzQYIpXALVS83NTPr8LLFK4LFsu',
+ 'anvato_mcp_anv_web_prod_791407490f4c1ef2a4bcb21103e0cb1bcb3352b3': 'rMOUZqe9lwcGq2mNgG3EDusm6lKgsUnczoOX3mbg',
+ 'anvato_mcp_gray_web_prod_4c10f067c393ed8fc453d3930f8ab2b159973900': 'rMOUZqe9lwcGq2mNgG3EDusm6lKgsUnczoOX3mbg',
+ 'anvato_mcp_hearst_web_prod_5356c3de0fc7c90a3727b4863ca7fec3a4524a99': 'P3uXJ0fXXditBPCGkfvlnVScpPEfKmc64Zv7ZgbK',
+ 'anvato_mcp_cbs_web_prod_02f26581ff80e5bda7aad28226a8d369037f2cbe': 'mGPvo5ZA5SgjOFAPEPXv7AnOpFUICX8hvFQVz69n',
+ 'anvato_mcp_telemundo_web_prod_c5278d51ad46fda4b6ca3d0ea44a7846a054f582': 'qyT6PXXLjVNCrHaRVj0ugAhalNRS7Ee9BP7LUokD',
+ 'nbcu_nbcd_watchvodv4_web_stage_4108362fba2d4ede21f262fea3c4162cbafd66c7': 'DhaU5lj0W2gEdcSSsnxURq8t7KIWtJfD966crVDk',
+ 'anvato_scripps_app_ios_prod_409c41960c60b308db43c3cc1da79cab9f1c3d93': 'WPxj5GraLTkYCyj3M7RozLqIycjrXOEcDGFMIJPn',
+ 'EZqvRyKBJLrgpClDPDF8I7Xpdp40Vx73': '4OxGd2dEakylntVKjKF0UK9PDPYB6A9W',
+ 'M2v78QkpleXm9hPp9jUXI63x5vA6BogR': 'ka6K32k7ZALmpINkjJUGUo0OE42Md1BQ',
+ 'nbcu_nbcd_desktop_web_prod_93d8ead38ce2024f8f544b78306fbd15895ae5e6_secure': 'NNemUkySjxLyPTKvZRiGntBIjEyK8uqicjMakIaQ'
+ }
+
+ _AUTH_KEY = b'\x31\xc2\x42\x84\x9e\x73\xa0\xce'
+
+ def __init__(self, *args, **kwargs):
+ super(AnvatoIE, self).__init__(*args, **kwargs)
+ self.__server_time = None
+
+ def _server_time(self, access_key, video_id):
+ if self.__server_time is not None:
+ return self.__server_time
+
+ self.__server_time = int(self._download_json(
+ self._api_prefix(access_key) + 'server_time?anvack=' + access_key, video_id,
+ note='Fetching server time')['server_time'])
+
+ return self.__server_time
+
+ def _api_prefix(self, access_key):
+ return 'https://tkx2-%s.anvato.net/rest/v2/' % ('prod' if 'prod' in access_key else 'stage')
+
+ def _get_video_json(self, access_key, video_id):
+ # See et() in anvplayer.min.js, which is an alias of getVideoJSON()
+ video_data_url = self._api_prefix(access_key) + 'mcp/video/%s?anvack=%s' % (video_id, access_key)
+ server_time = self._server_time(access_key, video_id)
+ input_data = '%d~%s~%s' % (server_time, md5_text(video_data_url), md5_text(server_time))
+
+ auth_secret = intlist_to_bytes(aes_encrypt(
+ bytes_to_intlist(input_data[:64]), bytes_to_intlist(self._AUTH_KEY)))
+
+ video_data_url += '&X-Anvato-Adst-Auth=' + base64.b64encode(auth_secret).decode('ascii')
+ anvrid = md5_text(time.time() * 1000 * random.random())[:30]
+ payload = {
+ 'api': {
+ 'anvrid': anvrid,
+ 'anvstk': md5_text('%s|%s|%d|%s' % (
+ access_key, anvrid, server_time, self._ANVACK_TABLE[access_key])),
+ 'anvts': server_time,
+ },
+ }
+
+ return self._download_json(
+ video_data_url, video_id, transform_source=strip_jsonp,
+ data=json.dumps(payload).encode('utf-8'))
+
+ def _extract_anvato_videos(self, webpage, video_id):
+ anvplayer_data = self._parse_json(self._html_search_regex(
+ r'<script[^>]+data-anvp=\'([^\']+)\'', webpage,
+ 'Anvato player data'), video_id)
+
+ video_id = anvplayer_data['video']
+ access_key = anvplayer_data['accessKey']
+
+ video_data = self._get_video_json(access_key, video_id)
+
+ formats = []
+ for published_url in video_data['published_urls']:
+ video_url = published_url['embed_url']
+ ext = determine_ext(video_url)
+
+ if ext == 'smil':
+ formats.extend(self._extract_smil_formats(video_url, video_id))
+ continue
+
+ tbr = int_or_none(published_url.get('kbps'))
+ a_format = {
+ 'url': video_url,
+ 'format_id': ('-'.join(filter(None, ['http', published_url.get('cdn_name')]))).lower(),
+ 'tbr': tbr if tbr != 0 else None,
+ }
+
+ if ext == 'm3u8':
+ # Not using _extract_m3u8_formats here as individual media
+ # playlists are also included in published_urls.
+ if tbr is None:
+ formats.append(self._m3u8_meta_format(video_url, ext='mp4', m3u8_id='hls'))
+ continue
+ else:
+ a_format.update({
+ 'format_id': '-'.join(filter(None, ['hls', compat_str(tbr)])),
+ 'ext': 'mp4',
+ })
+ elif ext == 'mp3':
+ a_format['vcodec'] = 'none'
+ else:
+ a_format.update({
+ 'width': int_or_none(published_url.get('width')),
+ 'height': int_or_none(published_url.get('height')),
+ })
+ formats.append(a_format)
+
+ self._sort_formats(formats)
+
+ subtitles = {}
+ for caption in video_data.get('captions', []):
+ a_caption = {
+ 'url': caption['url'],
+ 'ext': 'tt' if caption.get('format') == 'SMPTE-TT' else None
+ }
+ subtitles.setdefault(caption['language'], []).append(a_caption)
+
+ return {
+ 'id': video_id,
+ 'formats': formats,
+ 'title': video_data.get('def_title'),
+ 'description': video_data.get('def_description'),
+ 'categories': video_data.get('categories'),
+ 'thumbnail': video_data.get('thumbnail'),
+ 'subtitles': subtitles,
+ }
class AolIE(InfoExtractor):
IE_NAME = 'on.aol.com'
- _VALID_URL = r'(?:aol-video:|https?://on\.aol\.com/.*-)(?P<id>[^/?-]+)'
+ _VALID_URL = r'(?:aol-video:|https?://on\.aol\.com/(?:[^/]+/)*(?:[^/?#&]+-)?)(?P<id>[^/?#&]+)'
_TESTS = [{
# video with 5min ID
}, {
'url': 'http://on.aol.com/shows/park-bench-shw518173474-559a1b9be4b0c3bfad3357a7?context=SH:SHW518173474:PL4327:1460619712763',
'only_matching': True,
+ }, {
+ 'url': 'http://on.aol.com/video/519442220',
+ 'only_matching': True,
+ }, {
+ 'url': 'aol-video:5707d6b8e4b090497b04f706',
+ 'only_matching': True,
}]
def _real_extract(self, url):
}
-class ArteTVPlus7IE(InfoExtractor):
- IE_NAME = 'arte.tv:+7'
- _VALID_URL = r'https?://(?:www\.)?arte\.tv/guide/(?P<lang>fr|de|en|es)/(?:(?:sendungen|emissions|embed)/)?(?P<id>[^/]+)/(?P<name>[^/?#&+])'
-
+class ArteTVBaseIE(InfoExtractor):
@classmethod
def _extract_url_info(cls, url):
mobj = re.match(cls._VALID_URL, url)
video_id = mobj.group('id')
return video_id, lang
- def _real_extract(self, url):
- video_id, lang = self._extract_url_info(url)
- webpage = self._download_webpage(url, video_id)
- return self._extract_from_webpage(webpage, video_id, lang)
-
- def _extract_from_webpage(self, webpage, video_id, lang):
- patterns_templates = (r'arte_vp_url=["\'](.*?%s.*?)["\']', r'data-url=["\']([^"]+%s[^"]+)["\']')
- ids = (video_id, '')
- # some pages contain multiple videos (like
- # http://www.arte.tv/guide/de/sendungen/XEN/xenius/?vid=055918-015_PLUS7-D),
- # so we first try to look for json URLs that contain the video id from
- # the 'vid' parameter.
- patterns = [t % re.escape(_id) for _id in ids for t in patterns_templates]
- json_url = self._html_search_regex(
- patterns, webpage, 'json vp url', default=None)
- if not json_url:
- def find_iframe_url(webpage, default=NO_DEFAULT):
- return self._html_search_regex(
- r'<iframe[^>]+src=(["\'])(?P<url>.+\bjson_url=.+?)\1',
- webpage, 'iframe url', group='url', default=default)
-
- iframe_url = find_iframe_url(webpage, None)
- if not iframe_url:
- embed_url = self._html_search_regex(
- r'arte_vp_url_oembed=\'([^\']+?)\'', webpage, 'embed url', default=None)
- if embed_url:
- player = self._download_json(
- embed_url, video_id, 'Downloading player page')
- iframe_url = find_iframe_url(player['html'])
- # en and es URLs produce react-based pages with different layout (e.g.
- # http://www.arte.tv/guide/en/053330-002-A/carnival-italy?zone=world)
- if not iframe_url:
- program = self._search_regex(
- r'program\s*:\s*({.+?["\']embed_html["\'].+?}),?\s*\n',
- webpage, 'program', default=None)
- if program:
- embed_html = self._parse_json(program, video_id)
- if embed_html:
- iframe_url = find_iframe_url(embed_html['embed_html'])
- if iframe_url:
- json_url = compat_parse_qs(
- compat_urllib_parse_urlparse(iframe_url).query)['json_url'][0]
- if json_url:
- title = self._search_regex(
- r'<h3[^>]+title=(["\'])(?P<title>.+?)\1',
- webpage, 'title', default=None, group='title')
- return self._extract_from_json_url(json_url, video_id, lang, title=title)
- # Different kind of embed URL (e.g.
- # http://www.arte.tv/magazine/trepalium/fr/episode-0406-replay-trepalium)
- embed_url = self._search_regex(
- r'<iframe[^>]+src=(["\'])(?P<url>.+?)\1',
- webpage, 'embed url', group='url')
- return self.url_result(embed_url)
-
def _extract_from_json_url(self, json_url, video_id, lang, title=None):
info = self._download_json(json_url, video_id)
player_info = info['videoJsonPlayer']
'es': 'E[ESP]',
}
+ langcode = LANGS.get(lang, lang)
+
formats = []
for format_id, format_dict in player_info['VSR'].items():
f = dict(format_dict)
versionCode = f.get('versionCode')
- langcode = LANGS.get(lang, lang)
- lang_rexs = [r'VO?%s-' % re.escape(langcode), r'VO?.-ST%s$' % re.escape(langcode)]
- lang_pref = None
- if versionCode:
- matched_lang_rexs = [r for r in lang_rexs if re.match(r, versionCode)]
- lang_pref = -10 if not matched_lang_rexs else 10 * len(matched_lang_rexs)
- source_pref = 0
- if versionCode is not None:
- # The original version with subtitles has lower relevance
- if re.match(r'VO-ST(F|A|E)', versionCode):
- source_pref -= 10
- # The version with sourds/mal subtitles has also lower relevance
- elif re.match(r'VO?(F|A|E)-STM\1', versionCode):
- source_pref -= 9
+ l = re.escape(langcode)
+
+ # Language preference from most to least priority
+ # Reference: section 5.6.3 of
+ # http://www.arte.tv/sites/en/corporate/files/complete-technical-guidelines-arte-geie-v1-05.pdf
+ PREFERENCES = (
+ # original version in requested language, without subtitles
+ r'VO{0}$'.format(l),
+ # original version in requested language, with partial subtitles in requested language
+ r'VO{0}-ST{0}$'.format(l),
+ # original version in requested language, with subtitles for the deaf and hard-of-hearing in requested language
+ r'VO{0}-STM{0}$'.format(l),
+ # non-original (dubbed) version in requested language, without subtitles
+ r'V{0}$'.format(l),
+ # non-original (dubbed) version in requested language, with subtitles partial subtitles in requested language
+ r'V{0}-ST{0}$'.format(l),
+ # non-original (dubbed) version in requested language, with subtitles for the deaf and hard-of-hearing in requested language
+ r'V{0}-STM{0}$'.format(l),
+ # original version in requested language, with partial subtitles in different language
+ r'VO{0}-ST(?!{0}).+?$'.format(l),
+ # original version in requested language, with subtitles for the deaf and hard-of-hearing in different language
+ r'VO{0}-STM(?!{0}).+?$'.format(l),
+ # original version in different language, with partial subtitles in requested language
+ r'VO(?:(?!{0}).+?)?-ST{0}$'.format(l),
+ # original version in different language, with subtitles for the deaf and hard-of-hearing in requested language
+ r'VO(?:(?!{0}).+?)?-STM{0}$'.format(l),
+ # original version in different language, without subtitles
+ r'VO(?:(?!{0}))?$'.format(l),
+ # original version in different language, with partial subtitles in different language
+ r'VO(?:(?!{0}).+?)?-ST(?!{0}).+?$'.format(l),
+ # original version in different language, with subtitles for the deaf and hard-of-hearing in different language
+ r'VO(?:(?!{0}).+?)?-STM(?!{0}).+?$'.format(l),
+ )
+
+ for pref, p in enumerate(PREFERENCES):
+ if re.match(p, versionCode):
+ lang_pref = len(PREFERENCES) - pref
+ break
+ else:
+ lang_pref = -1
+
format = {
'format_id': format_id,
'preference': -10 if f.get('videoFormat') == 'M3U8' else None,
'height': int_or_none(f.get('height')),
'tbr': int_or_none(f.get('bitrate')),
'quality': qfunc(f.get('quality')),
- 'source_preference': source_pref,
}
if f.get('mediaType') == 'rtmp':
return info_dict
+class ArteTVPlus7IE(ArteTVBaseIE):
+ IE_NAME = 'arte.tv:+7'
+ _VALID_URL = r'https?://(?:www\.)?arte\.tv/guide/(?P<lang>fr|de|en|es)/(?:(?:sendungen|emissions|embed)/)?(?P<id>[^/]+)/(?P<name>[^/?#&]+)'
+
+ _TESTS = [{
+ 'url': 'http://www.arte.tv/guide/de/sendungen/XEN/xenius/?vid=055918-015_PLUS7-D',
+ 'only_matching': True,
+ }]
+
+ @classmethod
+ def suitable(cls, url):
+ return False if ArteTVPlaylistIE.suitable(url) else super(ArteTVPlus7IE, cls).suitable(url)
+
+ def _real_extract(self, url):
+ video_id, lang = self._extract_url_info(url)
+ webpage = self._download_webpage(url, video_id)
+ return self._extract_from_webpage(webpage, video_id, lang)
+
+ def _extract_from_webpage(self, webpage, video_id, lang):
+ patterns_templates = (r'arte_vp_url=["\'](.*?%s.*?)["\']', r'data-url=["\']([^"]+%s[^"]+)["\']')
+ ids = (video_id, '')
+ # some pages contain multiple videos (like
+ # http://www.arte.tv/guide/de/sendungen/XEN/xenius/?vid=055918-015_PLUS7-D),
+ # so we first try to look for json URLs that contain the video id from
+ # the 'vid' parameter.
+ patterns = [t % re.escape(_id) for _id in ids for t in patterns_templates]
+ json_url = self._html_search_regex(
+ patterns, webpage, 'json vp url', default=None)
+ if not json_url:
+ def find_iframe_url(webpage, default=NO_DEFAULT):
+ return self._html_search_regex(
+ r'<iframe[^>]+src=(["\'])(?P<url>.+\bjson_url=.+?)\1',
+ webpage, 'iframe url', group='url', default=default)
+
+ iframe_url = find_iframe_url(webpage, None)
+ if not iframe_url:
+ embed_url = self._html_search_regex(
+ r'arte_vp_url_oembed=\'([^\']+?)\'', webpage, 'embed url', default=None)
+ if embed_url:
+ player = self._download_json(
+ embed_url, video_id, 'Downloading player page')
+ iframe_url = find_iframe_url(player['html'])
+ # en and es URLs produce react-based pages with different layout (e.g.
+ # http://www.arte.tv/guide/en/053330-002-A/carnival-italy?zone=world)
+ if not iframe_url:
+ program = self._search_regex(
+ r'program\s*:\s*({.+?["\']embed_html["\'].+?}),?\s*\n',
+ webpage, 'program', default=None)
+ if program:
+ embed_html = self._parse_json(program, video_id)
+ if embed_html:
+ iframe_url = find_iframe_url(embed_html['embed_html'])
+ if iframe_url:
+ json_url = compat_parse_qs(
+ compat_urllib_parse_urlparse(iframe_url).query)['json_url'][0]
+ if json_url:
+ title = self._search_regex(
+ r'<h3[^>]+title=(["\'])(?P<title>.+?)\1',
+ webpage, 'title', default=None, group='title')
+ return self._extract_from_json_url(json_url, video_id, lang, title=title)
+ # Different kind of embed URL (e.g.
+ # http://www.arte.tv/magazine/trepalium/fr/episode-0406-replay-trepalium)
+ embed_url = self._search_regex(
+ r'<iframe[^>]+src=(["\'])(?P<url>.+?)\1',
+ webpage, 'embed url', group='url')
+ return self.url_result(embed_url)
+
+
# It also uses the arte_vp_url url from the webpage to extract the information
class ArteTVCreativeIE(ArteTVPlus7IE):
IE_NAME = 'arte.tv:creative'
IE_NAME = 'arte.tv:info'
_VALID_URL = r'https?://info\.arte\.tv/(?P<lang>fr|de|en|es)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
- _TEST = {
+ _TESTS = [{
'url': 'http://info.arte.tv/fr/service-civique-un-cache-misere',
'info_dict': {
'id': '067528-000-A',
'title': 'Service civique, un cache misère ?',
'upload_date': '20160403',
},
- }
+ }]
class ArteTVFutureIE(ArteTVPlus7IE):
IE_NAME = 'arte.tv:ddc'
_VALID_URL = r'https?://ddc\.arte\.tv/(?P<lang>emission|folge)/(?P<id>[^/?#&]+)'
+ _TESTS = []
+
def _real_extract(self, url):
video_id, lang = self._extract_url_info(url)
if lang == 'folge':
IE_NAME = 'arte.tv:concert'
_VALID_URL = r'https?://concert\.arte\.tv/(?P<lang>fr|de|en|es)/(?P<id>[^/?#&]+)'
- _TEST = {
+ _TESTS = [{
'url': 'http://concert.arte.tv/de/notwist-im-pariser-konzertclub-divan-du-monde',
'md5': '9ea035b7bd69696b67aa2ccaaa218161',
'info_dict': {
'upload_date': '20140128',
'description': 'md5:486eb08f991552ade77439fe6d82c305',
},
- }
+ }]
class ArteTVCinemaIE(ArteTVPlus7IE):
IE_NAME = 'arte.tv:cinema'
_VALID_URL = r'https?://cinema\.arte\.tv/(?P<lang>fr|de|en|es)/(?P<id>.+)'
- _TEST = {
+ _TESTS = [{
'url': 'http://cinema.arte.tv/de/node/38291',
'md5': '6b275511a5107c60bacbeeda368c3aa1',
'info_dict': {
'upload_date': '20160122',
'description': 'md5:7f749bbb77d800ef2be11d54529b96bc',
},
- }
+ }]
class ArteTVMagazineIE(ArteTVPlus7IE):
)
'''
+ _TESTS = []
+
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
lang = mobj.group('lang')
json_url = mobj.group('json_url')
return self._extract_from_json_url(json_url, video_id, lang)
+
+
+class ArteTVPlaylistIE(ArteTVBaseIE):
+ IE_NAME = 'arte.tv:playlist'
+ _VALID_URL = r'https?://(?:www\.)?arte\.tv/guide/(?P<lang>fr|de|en|es)/[^#]*#collection/(?P<id>PL-\d+)'
+
+ _TESTS = [{
+ 'url': 'http://www.arte.tv/guide/de/plus7/?country=DE#collection/PL-013263/ARTETV',
+ 'info_dict': {
+ 'id': 'PL-013263',
+ 'title': 'Areva & Uramin',
+ },
+ 'playlist_mincount': 6,
+ }, {
+ 'url': 'http://www.arte.tv/guide/de/playlists?country=DE#collection/PL-013190/ARTETV',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ playlist_id, lang = self._extract_url_info(url)
+ collection = self._download_json(
+ 'https://api.arte.tv/api/player/v1/collectionData/%s/%s?source=videos'
+ % (lang, playlist_id), playlist_id)
+ title = collection.get('title')
+ description = collection.get('shortDescription') or collection.get('teaserText')
+ entries = [
+ self._extract_from_json_url(
+ video['jsonUrl'], video.get('programId') or playlist_id, lang)
+ for video in collection['videos'] if video.get('jsonUrl')]
+ return self.playlist_result(entries, playlist_id, title, description)
'_skip': 'There is a limit of 200 free downloads / month for the test song'
}, {
'url': 'http://benprunty.bandcamp.com/track/lanius-battle',
- 'md5': '2b68e5851514c20efdff2afc5603b8b4',
+ 'md5': '73d0b3171568232574e45652f8720b5c',
'info_dict': {
'id': '2650410135',
'ext': 'mp3',
if m_trackinfo:
json_code = m_trackinfo.group(1)
data = json.loads(json_code)[0]
+ track_id = compat_str(data['id'])
+
+ if not data.get('file'):
+ raise ExtractorError('Not streamable', video_id=track_id, expected=True)
formats = []
for format_id, format_url in data['file'].items():
self._sort_formats(formats)
return {
- 'id': compat_str(data['id']),
+ 'id': track_id,
'title': data['title'],
'formats': formats,
'duration': float_or_none(data.get('duration')),
# coding: utf-8
from __future__ import unicode_literals
+import calendar
+import datetime
import re
from .common import InfoExtractor
-from ..compat import compat_str
+from ..compat import (
+ compat_etree_fromstring,
+ compat_str,
+ compat_parse_qs,
+ compat_xml_parse_error,
+)
from ..utils import (
- int_or_none,
- unescapeHTML,
ExtractorError,
+ int_or_none,
+ float_or_none,
xpath_text,
)
class BiliBiliIE(InfoExtractor):
- _VALID_URL = r'https?://www\.bilibili\.(?:tv|com)/video/av(?P<id>\d+)(?:/index_(?P<page_num>\d+).html)?'
+ _VALID_URL = r'https?://www\.bilibili\.(?:tv|com)/video/av(?P<id>\d+)'
_TESTS = [{
'url': 'http://www.bilibili.tv/video/av1074402/',
- 'md5': '2c301e4dab317596e837c3e7633e7d86',
+ 'md5': '5f7d29e1a2872f3df0cf76b1f87d3788',
'info_dict': {
'id': '1554319',
'ext': 'flv',
'title': '【金坷垃】金泡沫',
- 'duration': 308313,
+ 'description': 'md5:ce18c2a2d2193f0df2917d270f2e5923',
+ 'duration': 308.067,
+ 'timestamp': 1398012660,
'upload_date': '20140420',
'thumbnail': 're:^https?://.+\.jpg',
- 'description': 'md5:ce18c2a2d2193f0df2917d270f2e5923',
- 'timestamp': 1397983878,
'uploader': '菊子桑',
+ 'uploader_id': '156160',
},
}, {
'url': 'http://www.bilibili.com/video/av1041170/',
'id': '1041170',
'title': '【BD1080P】刀语【诸神&异域】',
'description': '这是个神奇的故事~每个人不留弹幕不给走哦~切利哦！~',
- 'uploader': '枫叶逝去',
- 'timestamp': 1396501299,
},
'playlist_count': 9,
+ }, {
+ 'url': 'http://www.bilibili.com/video/av4808130/',
+ 'info_dict': {
+ 'id': '4808130',
+ 'title': '【长篇】哆啦A梦443【钉铛】',
+ 'description': '(2016.05.27)来组合客人的脸吧&amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;illust_id=56912929',
+ },
+ 'playlist': [{
+ 'md5': '55cdadedf3254caaa0d5d27cf20a8f9c',
+ 'info_dict': {
+ 'id': '4808130_part1',
+ 'ext': 'flv',
+ 'title': '【长篇】哆啦A梦443【钉铛】',
+ 'description': '(2016.05.27)来组合客人的脸吧&amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;illust_id=56912929',
+ 'timestamp': 1464564180,
+ 'upload_date': '20160529',
+ 'uploader': '喜欢拉面',
+ 'uploader_id': '151066',
+ },
+ }, {
+ 'md5': '926f9f67d0c482091872fbd8eca7ea3d',
+ 'info_dict': {
+ 'id': '4808130_part2',
+ 'ext': 'flv',
+ 'title': '【长篇】哆啦A梦443【钉铛】',
+ 'description': '(2016.05.27)来组合客人的脸吧&amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;illust_id=56912929',
+ 'timestamp': 1464564180,
+ 'upload_date': '20160529',
+ 'uploader': '喜欢拉面',
+ 'uploader_id': '151066',
+ },
+ }, {
+ 'md5': '4b7b225b968402d7c32348c646f1fd83',
+ 'info_dict': {
+ 'id': '4808130_part3',
+ 'ext': 'flv',
+ 'title': '【长篇】哆啦A梦443【钉铛】',
+ 'description': '(2016.05.27)来组合客人的脸吧&amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;illust_id=56912929',
+ 'timestamp': 1464564180,
+ 'upload_date': '20160529',
+ 'uploader': '喜欢拉面',
+ 'uploader_id': '151066',
+ },
+ }, {
+ 'md5': '7b795e214166501e9141139eea236e91',
+ 'info_dict': {
+ 'id': '4808130_part4',
+ 'ext': 'flv',
+ 'title': '【长篇】哆啦A梦443【钉铛】',
+ 'description': '(2016.05.27)来组合客人的脸吧&amp;寻母六千里锭 抱歉，又轮到周日上班现在才到家 封面www.pixiv.net/member_illust.php?mode=medium&amp;illust_id=56912929',
+ 'timestamp': 1464564180,
+ 'upload_date': '20160529',
+ 'uploader': '喜欢拉面',
+ 'uploader_id': '151066',
+ },
+ }],
}]
+ # BiliBili blocks keys from time to time. The current key is extracted from
+ # the Android client
+ # TODO: find the sign algorithm used in the flash player
+ _APP_KEY = '86385cdc024c0f6c'
+
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('id')
- page_num = mobj.group('page_num') or '1'
- view_data = self._download_json(
- 'http://api.bilibili.com/view?type=json&appkey=8e9fc618fbd41e28&id=%s&page=%s' % (video_id, page_num),
- video_id)
- if 'error' in view_data:
- raise ExtractorError('%s said: %s' % (self.IE_NAME, view_data['error']), expected=True)
+ webpage = self._download_webpage(url, video_id)
- cid = view_data['cid']
- title = unescapeHTML(view_data['title'])
+ params = compat_parse_qs(self._search_regex(
+ [r'EmbedPlayer\([^)]+,\s*"([^"]+)"\)',
+ r'<iframe[^>]+src="https://secure\.bilibili\.com/secure,([^"]+)"'],
+ webpage, 'player parameters'))
+ cid = params['cid'][0]
- doc = self._download_xml(
- 'http://interface.bilibili.com/v_cdn_play?appkey=8e9fc618fbd41e28&cid=%s' % cid,
- cid,
- 'Downloading page %s/%s' % (page_num, view_data['pages'])
- )
+ info_xml_str = self._download_webpage(
+ 'http://interface.bilibili.com/v_cdn_play',
+ cid, query={'appkey': self._APP_KEY, 'cid': cid},
+ note='Downloading video info page')
- if xpath_text(doc, './result') == 'error':
- raise ExtractorError('%s said: %s' % (self.IE_NAME, xpath_text(doc, './message')), expected=True)
+ err_msg = None
+ durls = None
+ info_xml = None
+ try:
+ info_xml = compat_etree_fromstring(info_xml_str.encode('utf-8'))
+ except compat_xml_parse_error:
+ info_json = self._parse_json(info_xml_str, video_id, fatal=False)
+ err_msg = (info_json or {}).get('error_text')
+ else:
+ err_msg = xpath_text(info_xml, './message')
+
+ if info_xml is not None:
+ durls = info_xml.findall('./durl')
+ if not durls:
+ if err_msg:
+ raise ExtractorError('%s said: %s' % (self.IE_NAME, err_msg), expected=True)
+ else:
+ raise ExtractorError('No videos found!')
entries = []
- for durl in doc.findall('./durl'):
+ for durl in durls:
size = xpath_text(durl, ['./filesize', './size'])
formats = [{
'url': durl.find('./url').text,
'filesize': int_or_none(size),
- 'ext': 'flv',
}]
- backup_urls = durl.find('./backup_url')
- if backup_urls is not None:
- for backup_url in backup_urls.findall('./url'):
- formats.append({'url': backup_url.text})
- formats.reverse()
+ for backup_url in durl.findall('./backup_url/url'):
+ formats.append({
+ 'url': backup_url.text,
+ # backup URLs have lower priorities
+ 'preference': -2 if 'hd.mp4' in backup_url.text else -3,
+ })
+
+ self._sort_formats(formats)
entries.append({
'id': '%s_part%s' % (cid, xpath_text(durl, './order')),
- 'title': title,
'duration': int_or_none(xpath_text(durl, './length'), 1000),
'formats': formats,
})
+ title = self._html_search_regex('<h1[^>]+title="([^"]+)">', webpage, 'title')
+ description = self._html_search_meta('description', webpage)
+ datetime_str = self._html_search_regex(
+ r'<time[^>]+datetime="([^"]+)"', webpage, 'upload time', fatal=False)
+ if datetime_str:
+ timestamp = calendar.timegm(datetime.datetime.strptime(datetime_str, '%Y-%m-%dT%H:%M').timetuple())
+
+ # TODO 'view_count' requires deobfuscating Javascript
info = {
'id': compat_str(cid),
'title': title,
- 'description': view_data.get('description'),
- 'thumbnail': view_data.get('pic'),
- 'uploader': view_data.get('author'),
- 'timestamp': int_or_none(view_data.get('created')),
- 'view_count': int_or_none(view_data.get('play')),
- 'duration': int_or_none(xpath_text(doc, './timelength')),
+ 'description': description,
+ 'timestamp': timestamp,
+ 'thumbnail': self._html_search_meta('thumbnailUrl', webpage),
+ 'duration': float_or_none(xpath_text(info_xml, './timelength'), scale=1000),
}
+ uploader_mobj = re.search(
+ r'<a[^>]+href="https?://space\.bilibili\.com/(?P<id>\d+)"[^>]+title="(?P<name>[^"]+)"',
+ webpage)
+ if uploader_mobj:
+ info.update({
+ 'uploader': uploader_mobj.group('name'),
+ 'uploader_id': uploader_mobj.group('id'),
+ })
+
+ for entry in entries:
+ entry.update(info)
+
if len(entries) == 1:
- entries[0].update(info)
return entries[0]
else:
- info.update({
+ for idx, entry in enumerate(entries):
+ entry['id'] = '%s_part%d' % (video_id, (idx + 1))
+
+ return {
'_type': 'multi_video',
'id': video_id,
+ 'title': title,
+ 'description': description,
'entries': entries,
- })
- return info
+ }
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class BIQLEIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?biqle\.(?:com|org|ru)/watch/(?P<id>-?\d+_\d+)'
+ _TESTS = [{
+ 'url': 'http://www.biqle.ru/watch/847655_160197695',
+ 'md5': 'ad5f746a874ccded7b8f211aeea96637',
+ 'info_dict': {
+ 'id': '160197695',
+ 'ext': 'mp4',
+ 'title': 'Foo Fighters - The Pretender (Live at Wembley Stadium)',
+ 'uploader': 'Andrey Rogozin',
+ 'upload_date': '20110605',
+ }
+ }, {
+ 'url': 'https://biqle.org/watch/-44781847_168547604',
+ 'md5': '7f24e72af1db0edf7c1aaba513174f97',
+ 'info_dict': {
+ 'id': '168547604',
+ 'ext': 'mp4',
+ 'title': 'Ребенок в шоке от автоматической мойки',
+ 'uploader': 'Dmitry Kotov',
+ }
+ }]
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ webpage = self._download_webpage(url, video_id)
+ embed_url = self._proto_relative_url(self._search_regex(
+ r'<iframe.+?src="((?:http:)?//daxab\.com/[^"]+)".*?></iframe>', webpage, 'embed url'))
+
+ return {
+ '_type': 'url_transparent',
+ 'url': embed_url,
+ }
'title': 'Shah\'s Presentation on Foreign-Exchange Strategies',
'description': 'md5:a8ba0302912d03d246979735c17d2761',
},
+ 'params': {
+ 'format': 'best[format_id^=hds]',
+ },
}, {
'url': 'http://www.bloomberg.com/news/articles/2015-11-12/five-strange-things-that-have-been-happening-in-financial-markets',
'only_matching': True,
playlist_title=playlist_info['mediaCollectionDTO']['displayName'])
def _extract_video_info(self, video_info):
+ video_id = compat_str(video_info['id'])
publisher_id = video_info.get('publisherId')
info = {
- 'id': compat_str(video_info['id']),
+ 'id': video_id,
'title': video_info['displayName'].strip(),
'description': video_info.get('shortDescription'),
'thumbnail': video_info.get('videoStillURL') or video_info.get('thumbnailURL'),
url_comp = compat_urllib_parse_urlparse(url)
if url_comp.path.endswith('.m3u8'):
formats.extend(
- self._extract_m3u8_formats(url, info['id'], 'mp4'))
+ self._extract_m3u8_formats(
+ url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
continue
elif 'akamaihd.net' in url_comp.netloc:
# This type of renditions are served through
a_format.update({
'format_id': 'hls%s' % ('-%s' % tbr if tbr else ''),
'ext': 'mp4',
- 'protocol': 'm3u8',
+ 'protocol': 'm3u8_native',
})
formats.append(a_format)
return ad_info
if 'url' not in info and not info.get('formats'):
- raise ExtractorError('Unable to extract video url for %s' % info['id'])
+ raise ExtractorError('Unable to extract video url for %s' % video_id)
return info
# non numeric ref: prefixed video id
'url': 'http://players.brightcove.net/710858724001/default_default/index.html?videoId=ref:event-stream-356',
'only_matching': True,
+ }, {
+ # unavailable video without message but with error_code
+ 'url': 'http://players.brightcove.net/1305187701/c832abfb-641b-44eb-9da0-2fe76786505f_default/index.html?videoId=4377407326001',
+ 'only_matching': True,
}]
@staticmethod
})
except ExtractorError as e:
if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
- json_data = self._parse_json(e.cause.read().decode(), video_id)
- raise ExtractorError(json_data[0]['message'], expected=True)
+ json_data = self._parse_json(e.cause.read().decode(), video_id)[0]
+ raise ExtractorError(
+ json_data.get('message') or json_data['error_code'], expected=True)
raise
title = json_data['name'].strip()
if not src:
continue
formats.extend(self._extract_m3u8_formats(
- src, video_id, 'mp4', m3u8_id='hls', fatal=False))
+ src, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
elif source_type == 'application/dash+xml':
if not src:
continue
_VALID_URL = r'^https?://(?:www\.)?byutv.org/watch/[0-9a-f-]+/(?P<video_id>[^/?#]+)'
_TEST = {
'url': 'http://www.byutv.org/watch/6587b9a3-89d2-42a6-a7f7-fd2f81840a7d/studio-c-season-5-episode-5',
+ 'md5': '05850eb8c749e2ee05ad5a1c34668493',
'info_dict': {
'id': 'studio-c-season-5-episode-5',
'ext': 'mp4',
},
'params': {
'skip_download': True,
- }
+ },
+ 'add_ie': ['Ooyala'],
}
def _real_extract(self, url):
import re
from .common import InfoExtractor
-from ..utils import js_to_json
+from ..utils import (
+ js_to_json,
+ smuggle_url,
+)
class CBCIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?cbc\.ca/(?:[^/]+/)+(?P<id>[^/?#]+)'
+ _VALID_URL = r'https?://(?:www\.)?cbc\.ca/(?!player/)(?:[^/]+/)+(?P<id>[^/?#]+)'
_TESTS = [{
# with mediaId
'url': 'http://www.cbc.ca/22minutes/videos/clips-season-23/don-cherry-play-offs',
+ 'md5': '97e24d09672fc4cf56256d6faa6c25bc',
'info_dict': {
'id': '2682904050',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'Don Cherry – All-Stars',
'description': 'Don Cherry has a bee in his bonnet about AHL player John Scott because that guy’s got heart.',
- 'timestamp': 1454475540,
+ 'timestamp': 1454463000,
'upload_date': '20160203',
- },
- 'params': {
- # rtmp download
- 'skip_download': True,
+ 'uploader': 'CBCC-NEW',
},
}, {
# with clipId
'url': 'http://www.cbc.ca/archives/entry/1978-robin-williams-freestyles-on-90-minutes-live',
+ 'md5': '0274a90b51a9b4971fe005c63f592f12',
'info_dict': {
'id': '2487345465',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'Robin Williams freestyles on 90 Minutes Live',
'description': 'Wacky American comedian Robin Williams shows off his infamous "freestyle" comedic talents while being interviewed on CBC\'s 90 Minutes Live.',
- 'upload_date': '19700101',
- },
- 'params': {
- # rtmp download
- 'skip_download': True,
+ 'upload_date': '19780210',
+ 'uploader': 'CBCC-NEW',
+ 'timestamp': 255977160,
},
}, {
# multiple iframes
'url': 'http://www.cbc.ca/natureofthings/blog/birds-eye-view-from-vancouvers-burrard-street-bridge-how-we-got-the-shot',
'playlist': [{
+ 'md5': '377572d0b49c4ce0c9ad77470e0b96b4',
'info_dict': {
'id': '2680832926',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'An Eagle\'s-Eye View Off Burrard Bridge',
'description': 'Hercules the eagle flies from Vancouver\'s Burrard Bridge down to a nearby park with a mini-camera strapped to his back.',
- 'upload_date': '19700101',
+ 'upload_date': '20160201',
+ 'timestamp': 1454342820,
+ 'uploader': 'CBCC-NEW',
},
}, {
+ 'md5': '415a0e3f586113894174dfb31aa5bb1a',
'info_dict': {
'id': '2658915080',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'Fly like an eagle!',
'description': 'Eagle equipped with a mini camera flies from the world\'s tallest tower',
- 'upload_date': '19700101',
+ 'upload_date': '20150315',
+ 'timestamp': 1426443984,
+ 'uploader': 'CBCC-NEW',
},
}],
- 'params': {
- # rtmp download
- 'skip_download': True,
- },
}]
@classmethod
class CBCPlayerIE(InfoExtractor):
_VALID_URL = r'(?:cbcplayer:|https?://(?:www\.)?cbc\.ca/(?:player/play/|i/caffeine/syndicate/\?mediaId=))(?P<id>\d+)'
- _TEST = {
+ _TESTS = [{
'url': 'http://www.cbc.ca/player/play/2683190193',
+ 'md5': '64d25f841ddf4ddb28a235338af32e2c',
'info_dict': {
'id': '2683190193',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'Gerry Runs a Sweat Shop',
'description': 'md5:b457e1c01e8ff408d9d801c1c2cd29b0',
- 'timestamp': 1455067800,
+ 'timestamp': 1455071400,
'upload_date': '20160210',
+ 'uploader': 'CBCC-NEW',
},
- 'params': {
- # rtmp download
- 'skip_download': True,
+ }, {
+ # Redirected from http://www.cbc.ca/player/AudioMobile/All%20in%20a%20Weekend%20Montreal/ID/2657632011/
+ 'url': 'http://www.cbc.ca/player/play/2657631896',
+ 'md5': 'e5e708c34ae6fca156aafe17c43e8b75',
+ 'info_dict': {
+ 'id': '2657631896',
+ 'ext': 'mp3',
+ 'title': 'CBC Montreal is organizing its first ever community hackathon!',
+ 'description': 'The modern technology we tend to depend on so heavily, is never without it\'s share of hiccups and headaches. Next weekend - CBC Montreal will be getting members of the public for its first Hackathon.',
+ 'timestamp': 1425704400,
+ 'upload_date': '20150307',
+ 'uploader': 'CBCC-NEW',
},
- }
+ }, {
+ # available only when we add `formats=MPEG4,FLV,MP3` to theplatform url
+ 'url': 'http://www.cbc.ca/player/play/2164402062',
+ 'md5': '17a61eb813539abea40618d6323a7f82',
+ 'info_dict': {
+ 'id': '2164402062',
+ 'ext': 'flv',
+ 'title': 'Cancer survivor four times over',
+ 'description': 'Tim Mayer has beaten three different forms of cancer four times in five years.',
+ 'timestamp': 1320410746,
+ 'upload_date': '20111104',
+ 'uploader': 'CBCC-NEW',
+ },
+ }]
def _real_extract(self, url):
video_id = self._match_id(url)
- return self.url_result(
- 'http://feed.theplatform.com/f/ExhSPC/vms_5akSXx4Ng_Zn?byGuid=%s' % video_id,
- 'ThePlatformFeed', video_id)
+ return {
+ '_type': 'url_transparent',
+ 'ie_key': 'ThePlatform',
+ 'url': smuggle_url(
+ 'http://link.theplatform.com/s/ExhSPC/media/guid/2655402169/%s?mbr=true&formats=MPEG4,FLV,MP3' % video_id, {
+ 'force_smil_url': True
+ }),
+ 'id': video_id,
+ }
from __future__ import unicode_literals
+import re
+
from .theplatform import ThePlatformIE
from ..utils import (
xpath_text,
class CBSIE(CBSBaseIE):
- _VALID_URL = r'https?://(?:www\.)?(?:cbs\.com/shows/[^/]+/(?:video|artist)|colbertlateshow\.com/(?:video|podcasts))/[^/]+/(?P<id>[^/]+)'
+ _VALID_URL = r'(?:cbs:(?P<content_id>\w+)|https?://(?:www\.)?(?:cbs\.com/shows/[^/]+/(?:video|artist)|colbertlateshow\.com/(?:video|podcasts))/[^/]+/(?P<display_id>[^/]+))'
_TESTS = [{
'url': 'http://www.cbs.com/shows/garth-brooks/video/_u7W953k6la293J7EPTd9oHkSPs6Xn6_/connect-chat-feat-garth-brooks/',
TP_RELEASE_URL_TEMPLATE = 'http://link.theplatform.com/s/dJ5BDC/%s?mbr=true'
def _real_extract(self, url):
- display_id = self._match_id(url)
- webpage = self._download_webpage(url, display_id)
- content_id = self._search_regex(
- [r"video\.settings\.content_id\s*=\s*'([^']+)';", r"cbsplayer\.contentId\s*=\s*'([^']+)';"],
- webpage, 'content id')
+ content_id, display_id = re.match(self._VALID_URL, url).groups()
+ if not content_id:
+ webpage = self._download_webpage(url, display_id)
+ content_id = self._search_regex(
+ [r"video\.settings\.content_id\s*=\s*'([^']+)';", r"cbsplayer\.contentId\s*=\s*'([^']+)';"],
+ webpage, 'content id')
items_data = self._download_xml(
'http://can.cbs.com/thunder/player/videoPlayerService.php',
content_id, query={'partner': 'cbs', 'contentId': content_id})
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import calendar
+import datetime
+
+from .anvato import AnvatoIE
+from .sendtonews import SendtoNewsIE
+from ..compat import compat_urlparse
+
+
+class CBSLocalIE(AnvatoIE):
+ _VALID_URL = r'https?://[a-z]+\.cbslocal\.com/\d+/\d+/\d+/(?P<id>[0-9a-z-]+)'
+
+ _TESTS = [{
+ # Anvato backend
+ 'url': 'http://losangeles.cbslocal.com/2016/05/16/safety-advocates-say-fatal-car-seat-failures-are-public-health-crisis',
+ 'md5': 'f0ee3081e3843f575fccef901199b212',
+ 'info_dict': {
+ 'id': '3401037',
+ 'ext': 'mp4',
+ 'title': 'Safety Advocates Say Fatal Car Seat Failures Are \'Public Health Crisis\'',
+ 'description': 'Collapsing seats have been the focus of scrutiny for decades, though experts say remarkably little has been done to address the issue. Randy Paige reports.',
+ 'thumbnail': 're:^https?://.*',
+ 'timestamp': 1463440500,
+ 'upload_date': '20160516',
+ 'subtitles': {
+ 'en': 'mincount:5',
+ },
+ 'categories': [
+ 'Stations\\Spoken Word\\KCBSTV',
+ 'Syndication\\MSN',
+ 'Syndication\\NDN',
+ 'Syndication\\AOL',
+ 'Syndication\\Yahoo',
+ 'Syndication\\Tribune',
+ 'Syndication\\Curb.tv',
+ 'Content\\News'
+ ],
+ },
+ }, {
+ # SendtoNews embed
+ 'url': 'http://cleveland.cbslocal.com/2016/05/16/indians-score-season-high-15-runs-in-blowout-win-over-reds-rapid-reaction/',
+ 'info_dict': {
+ 'id': 'GxfCe0Zo7D-175909-5588',
+ 'ext': 'mp4',
+ 'title': 'Recap: CLE 15, CIN 6',
+ 'description': '5/16/16: Indians\' bats explode for 15 runs in a win',
+ 'upload_date': '20160516',
+ 'timestamp': 1463433840,
+ 'duration': 49,
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
+ }]
+
+ def _real_extract(self, url):
+ display_id = self._match_id(url)
+ webpage = self._download_webpage(url, display_id)
+
+ sendtonews_url = SendtoNewsIE._extract_url(webpage)
+ if sendtonews_url:
+ info_dict = {
+ '_type': 'url_transparent',
+ 'url': compat_urlparse.urljoin(url, sendtonews_url),
+ }
+ else:
+ info_dict = self._extract_anvato_videos(webpage, display_id)
+
+ time_str = self._html_search_regex(
+ r'class="entry-date">([^<]+)<', webpage, 'released date', fatal=False)
+ timestamp = None
+ if time_str:
+ timestamp = calendar.timegm(datetime.datetime.strptime(
+ time_str, '%b %d, %Y %I:%M %p').timetuple())
+
+ info_dict.update({
+ 'display_id': display_id,
+ 'timestamp': timestamp,
+ })
+
+ return info_dict
from __future__ import unicode_literals
-import re
-
from .common import InfoExtractor
from ..utils import (
int_or_none,
- parse_duration,
- qualities,
- unified_strdate,
+ parse_iso8601,
)
'url': 'https://media.ccc.de/v/30C3_-_5443_-_en_-_saal_g_-_201312281830_-_introduction_to_processor_design_-_byterazor#video',
'md5': '3a1eda8f3a29515d27f5adb967d7e740',
'info_dict': {
- 'id': '30C3_-_5443_-_en_-_saal_g_-_201312281830_-_introduction_to_processor_design_-_byterazor',
+ 'id': '1839',
'ext': 'mp4',
'title': 'Introduction to Processor Design',
- 'description': 'md5:80be298773966f66d56cb11260b879af',
+ 'description': 'md5:df55f6d073d4ceae55aae6f2fd98a0ac',
'thumbnail': 're:^https?://.*\.jpg$',
- 'view_count': int,
'upload_date': '20131228',
- 'duration': 3660,
+ 'timestamp': 1388188800,
+ 'duration': 3710,
}
}, {
'url': 'https://media.ccc.de/v/32c3-7368-shopshifting#download',
}]
def _real_extract(self, url):
- video_id = self._match_id(url)
- webpage = self._download_webpage(url, video_id)
-
- if self._downloader.params.get('prefer_free_formats'):
- preference = qualities(['mp3', 'opus', 'mp4-lq', 'webm-lq', 'h264-sd', 'mp4-sd', 'webm-sd', 'mp4', 'webm', 'mp4-hd', 'h264-hd', 'webm-hd'])
- else:
- preference = qualities(['opus', 'mp3', 'webm-lq', 'mp4-lq', 'webm-sd', 'h264-sd', 'mp4-sd', 'webm', 'mp4', 'webm-hd', 'mp4-hd', 'h264-hd'])
-
- title = self._html_search_regex(
- r'(?s)<h1>(.*?)</h1>', webpage, 'title')
- description = self._html_search_regex(
- r'(?s)<h3>About</h3>(.+?)<h3>',
- webpage, 'description', fatal=False)
- upload_date = unified_strdate(self._html_search_regex(
- r"(?s)<span[^>]+class='[^']*fa-calendar-o'[^>]*>(.+?)</span>",
- webpage, 'upload date', fatal=False))
- view_count = int_or_none(self._html_search_regex(
- r"(?s)<span class='[^']*fa-eye'></span>(.*?)</li>",
- webpage, 'view count', fatal=False))
- duration = parse_duration(self._html_search_regex(
- r'(?s)<span[^>]+class=(["\']).*?fa-clock-o.*?\1[^>]*></span>(?P<duration>.+?)</li',
- webpage, 'duration', fatal=False, group='duration'))
+ display_id = self._match_id(url)
+ webpage = self._download_webpage(url, display_id)
+ event_id = self._search_regex("data-id='(\d+)'", webpage, 'event id')
+ event_data = self._download_json('https://media.ccc.de/public/events/%s' % event_id, event_id)
- matches = re.finditer(r'''(?xs)
- <(?:span|div)\s+class='label\s+filetype'>(?P<format>[^<]*)</(?:span|div)>\s*
- <(?:span|div)\s+class='label\s+filetype'>(?P<lang>[^<]*)</(?:span|div)>\s*
- <a\s+download\s+href='(?P<http_url>[^']+)'>\s*
- (?:
- .*?
- <a\s+(?:download\s+)?href='(?P<torrent_url>[^']+\.torrent)'
- )?''', webpage)
formats = []
- for m in matches:
- format = m.group('format')
- format_id = self._search_regex(
- r'.*/([a-z0-9_-]+)/[^/]*$',
- m.group('http_url'), 'format id', default=None)
- if format_id:
- format_id = m.group('lang') + '-' + format_id
- vcodec = 'h264' if 'h264' in format_id else (
- 'none' if format_id in ('mp3', 'opus') else None
+ for recording in event_data.get('recordings', []):
+ recording_url = recording.get('recording_url')
+ if not recording_url:
+ continue
+ language = recording.get('language')
+ folder = recording.get('folder')
+ format_id = None
+ if language:
+ format_id = language
+ if folder:
+ if language:
+ format_id += '-' + folder
+ else:
+ format_id = folder
+ vcodec = 'h264' if 'h264' in folder else (
+ 'none' if folder in ('mp3', 'opus') else None
)
formats.append({
'format_id': format_id,
- 'format': format,
- 'language': m.group('lang'),
- 'url': m.group('http_url'),
+ 'url': recording_url,
+ 'width': int_or_none(recording.get('width')),
+ 'height': int_or_none(recording.get('height')),
+ 'filesize': int_or_none(recording.get('size'), invscale=1024 * 1024),
+ 'language': language,
'vcodec': vcodec,
- 'preference': preference(format_id),
})
-
- if m.group('torrent_url'):
- formats.append({
- 'format_id': 'torrent-%s' % (format if format_id is None else format_id),
- 'format': '%s (torrent)' % format,
- 'proto': 'torrent',
- 'format_note': '(unsupported; will just download the .torrent file)',
- 'vcodec': vcodec,
- 'preference': -100 + preference(format_id),
- 'url': m.group('torrent_url'),
- })
self._sort_formats(formats)
- thumbnail = self._html_search_regex(
- r"<video.*?poster='([^']+)'", webpage, 'thumbnail', fatal=False)
-
return {
- 'id': video_id,
- 'title': title,
- 'description': description,
- 'thumbnail': thumbnail,
- 'view_count': view_count,
- 'upload_date': upload_date,
- 'duration': duration,
+ 'id': event_id,
+ 'display_id': display_id,
+ 'title': event_data['title'],
+ 'description': event_data.get('description'),
+ 'thumbnail': event_data.get('thumb_url'),
+ 'timestamp': parse_iso8601(event_data.get('date')),
+ 'duration': int_or_none(event_data.get('length')),
+ 'tags': event_data.get('tags'),
'formats': formats,
}
'skip_download': True,
},
}, {
- 'url': 'http://www.ceskatelevize.cz/ivysilani/10532695142-prvni-republika/bonus/14716-zpevacka-z-duparny-bobina',
+ 'url': 'http://www.ceskatelevize.cz/ivysilani/10441294653-hyde-park-civilizace/215411058090502/bonus/20641-bonus-01-en',
'info_dict': {
- 'id': '61924494876844374',
+ 'id': '61924494877028507',
'ext': 'mp4',
- 'title': 'První republika: Zpěvačka z Dupárny Bobina',
- 'description': 'Sága mapující atmosféru první republiky od r. 1918 do r. 1945.',
+ 'title': 'Hyde Park Civilizace: Bonus 01 - En',
+ 'description': 'English Subtittles',
'thumbnail': 're:^https?://.*\.jpg',
- 'duration': 88.4,
+ 'duration': 81.3,
},
'params': {
# m3u8 download
'skip_download': True,
},
+ }, {
+ # live stream
+ 'url': 'http://www.ceskatelevize.cz/ivysilani/zive/ct4/',
+ 'info_dict': {
+ 'id': 402,
+ 'ext': 'mp4',
+ 'title': 're:^ČT Sport \d{4}-\d{2}-\d{2} \d{2}:\d{2}$',
+ 'is_live': True,
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
+ 'skip': 'Georestricted to Czech Republic',
}, {
# video with 18+ caution trailer
'url': 'http://www.ceskatelevize.cz/porady/10520528904-queer/215562210900007-bogotart/',
req = sanitized_Request(compat_urllib_parse_unquote(playlist_url))
req.add_header('Referer', url)
- playlist_title = self._og_search_title(webpage)
- playlist_description = self._og_search_description(webpage)
+ playlist_title = self._og_search_title(webpage, default=None)
+ playlist_description = self._og_search_description(webpage, default=None)
playlist = self._download_json(req, playlist_id)['playlist']
playlist_len = len(playlist)
entries = []
for item in playlist:
+ is_live = item.get('type') == 'LIVE'
formats = []
for format_id, stream_url in item['streamUrls'].items():
formats.extend(self._extract_m3u8_formats(
stream_url, playlist_id, 'mp4',
- entry_protocol='m3u8_native', fatal=False))
+ entry_protocol='m3u8' if is_live else 'm3u8_native',
+ fatal=False))
self._sort_formats(formats)
item_id = item.get('id') or item['assetId']
if subs:
subtitles = self.extract_subtitles(episode_id, subs)
+ if playlist_len == 1:
+ final_title = playlist_title or title
+ if is_live:
+ final_title = self._live_title(final_title)
+ else:
+ final_title = '%s (%s)' % (playlist_title, title)
+
entries.append({
'id': item_id,
- 'title': playlist_title if playlist_len == 1 else '%s (%s)' % (playlist_title, title),
+ 'title': final_title,
'description': playlist_description if playlist_len == 1 else None,
'thumbnail': thumbnail,
'duration': duration,
'formats': formats,
'subtitles': subtitles,
+ 'is_live': is_live,
})
return self.playlist_result(entries, playlist_id, playlist_title, playlist_description)
'''
IE_DESC = 'Channel 9'
IE_NAME = 'channel9'
- _VALID_URL = r'https?://(?:www\.)?channel9\.msdn\.com/(?P<contentpath>.+)/?'
-
- _TESTS = [
- {
- 'url': 'http://channel9.msdn.com/Events/TechEd/Australia/2013/KOS002',
- 'md5': 'bbd75296ba47916b754e73c3a4bbdf10',
- 'info_dict': {
- 'id': 'Events/TechEd/Australia/2013/KOS002',
- 'ext': 'mp4',
- 'title': 'Developer Kick-Off Session: Stuff We Love',
- 'description': 'md5:c08d72240b7c87fcecafe2692f80e35f',
- 'duration': 4576,
- 'thumbnail': 're:http://.*\.jpg',
- 'session_code': 'KOS002',
- 'session_day': 'Day 1',
- 'session_room': 'Arena 1A',
- 'session_speakers': ['Ed Blankenship', 'Andrew Coates', 'Brady Gaster', 'Patrick Klug', 'Mads Kristensen'],
- },
+ _VALID_URL = r'https?://(?:www\.)?channel9\.msdn\.com/(?P<contentpath>.+?)(?P<rss>/RSS)?/?(?:[?#&]|$)'
+
+ _TESTS = [{
+ 'url': 'http://channel9.msdn.com/Events/TechEd/Australia/2013/KOS002',
+ 'md5': 'bbd75296ba47916b754e73c3a4bbdf10',
+ 'info_dict': {
+ 'id': 'Events/TechEd/Australia/2013/KOS002',
+ 'ext': 'mp4',
+ 'title': 'Developer Kick-Off Session: Stuff We Love',
+ 'description': 'md5:c08d72240b7c87fcecafe2692f80e35f',
+ 'duration': 4576,
+ 'thumbnail': 're:http://.*\.jpg',
+ 'session_code': 'KOS002',
+ 'session_day': 'Day 1',
+ 'session_room': 'Arena 1A',
+ 'session_speakers': ['Ed Blankenship', 'Andrew Coates', 'Brady Gaster', 'Patrick Klug',
+ 'Mads Kristensen'],
},
- {
- 'url': 'http://channel9.msdn.com/posts/Self-service-BI-with-Power-BI-nuclear-testing',
- 'md5': 'b43ee4529d111bc37ba7ee4f34813e68',
- 'info_dict': {
- 'id': 'posts/Self-service-BI-with-Power-BI-nuclear-testing',
- 'ext': 'mp4',
- 'title': 'Self-service BI with Power BI - nuclear testing',
- 'description': 'md5:d1e6ecaafa7fb52a2cacdf9599829f5b',
- 'duration': 1540,
- 'thumbnail': 're:http://.*\.jpg',
- 'authors': ['Mike Wilmot'],
- },
+ }, {
+ 'url': 'http://channel9.msdn.com/posts/Self-service-BI-with-Power-BI-nuclear-testing',
+ 'md5': 'b43ee4529d111bc37ba7ee4f34813e68',
+ 'info_dict': {
+ 'id': 'posts/Self-service-BI-with-Power-BI-nuclear-testing',
+ 'ext': 'mp4',
+ 'title': 'Self-service BI with Power BI - nuclear testing',
+ 'description': 'md5:d1e6ecaafa7fb52a2cacdf9599829f5b',
+ 'duration': 1540,
+ 'thumbnail': 're:http://.*\.jpg',
+ 'authors': ['Mike Wilmot'],
},
- {
- # low quality mp4 is best
- 'url': 'https://channel9.msdn.com/Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
- 'info_dict': {
- 'id': 'Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
- 'ext': 'mp4',
- 'title': 'Ranges for the Standard Library',
- 'description': 'md5:2e6b4917677af3728c5f6d63784c4c5d',
- 'duration': 5646,
- 'thumbnail': 're:http://.*\.jpg',
- },
- 'params': {
- 'skip_download': True,
- },
- }
- ]
+ }, {
+ # low quality mp4 is best
+ 'url': 'https://channel9.msdn.com/Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
+ 'info_dict': {
+ 'id': 'Events/CPP/CppCon-2015/Ranges-for-the-Standard-Library',
+ 'ext': 'mp4',
+ 'title': 'Ranges for the Standard Library',
+ 'description': 'md5:2e6b4917677af3728c5f6d63784c4c5d',
+ 'duration': 5646,
+ 'thumbnail': 're:http://.*\.jpg',
+ },
+ 'params': {
+ 'skip_download': True,
+ },
+ }, {
+ 'url': 'https://channel9.msdn.com/Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b/RSS',
+ 'info_dict': {
+ 'id': 'Niners/Splendid22/Queue/76acff796e8f411184b008028e0d492b',
+ 'title': 'Channel 9',
+ },
+ 'playlist_count': 2,
+ }, {
+ 'url': 'https://channel9.msdn.com/Events/DEVintersection/DEVintersection-2016/RSS',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://channel9.msdn.com/Events/Speakers/scott-hanselman/RSS?UrlSafeName=scott-hanselman',
+ 'only_matching': True,
+ }]
_RSS_URL = 'http://channel9.msdn.com/%s/RSS'
return self.playlist_result(contents)
- def _extract_list(self, content_path):
- rss = self._download_xml(self._RSS_URL % content_path, content_path, 'Downloading RSS')
+ def _extract_list(self, video_id, rss_url=None):
+ if not rss_url:
+ rss_url = self._RSS_URL % video_id
+ rss = self._download_xml(rss_url, video_id, 'Downloading RSS')
entries = [self.url_result(session_url.text, 'Channel9')
for session_url in rss.findall('./channel/item/link')]
title_text = rss.find('./channel/title').text
- return self.playlist_result(entries, content_path, title_text)
+ return self.playlist_result(entries, video_id, title_text)
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
content_path = mobj.group('contentpath')
+ rss = mobj.group('rss')
+
+ if rss:
+ return self._extract_list(content_path, url)
- webpage = self._download_webpage(url, content_path, 'Downloading web page')
+ webpage = self._download_webpage(
+ url, content_path, 'Downloading web page')
- page_type_m = re.search(r'<meta name="WT.entryid" content="(?P<pagetype>[^:]+)[^"]+"/>', webpage)
- if page_type_m is not None:
- page_type = page_type_m.group('pagetype')
+ page_type = self._search_regex(
+ r'<meta[^>]+name=(["\'])WT\.entryid\1[^>]+content=(["\'])(?P<pagetype>[^:]+).+?\2',
+ webpage, 'page type', default=None, group='pagetype')
+ if page_type:
if page_type == 'Entry': # Any 'item'-like page, may contain downloadable content
return self._extract_entry_item(webpage, content_path)
elif page_type == 'Session': # Event session page, may contain downloadable content
return self._extract_list(content_path)
else:
raise ExtractorError('Unexpected WT.entryid %s' % page_type, expected=True)
-
else: # Assuming list
return self._extract_list(content_path)
+++ /dev/null
-# encoding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import ExtractorError
-from .screenwavemedia import ScreenwaveMediaIE
-
-
-class CinemassacreIE(InfoExtractor):
- _VALID_URL = 'https?://(?:www\.)?cinemassacre\.com/(?P<date_y>[0-9]{4})/(?P<date_m>[0-9]{2})/(?P<date_d>[0-9]{2})/(?P<display_id>[^?#/]+)'
- _TESTS = [
- {
- 'url': 'http://cinemassacre.com/2012/11/10/avgn-the-movie-trailer/',
- 'md5': 'fde81fbafaee331785f58cd6c0d46190',
- 'info_dict': {
- 'id': 'Cinemassacre-19911',
- 'ext': 'mp4',
- 'upload_date': '20121110',
- 'title': '“Angry Video Game Nerd: The Movie” – Trailer',
- 'description': 'md5:fb87405fcb42a331742a0dce2708560b',
- },
- 'params': {
- # m3u8 download
- 'skip_download': True,
- },
- },
- {
- 'url': 'http://cinemassacre.com/2013/10/02/the-mummys-hand-1940',
- 'md5': 'd72f10cd39eac4215048f62ab477a511',
- 'info_dict': {
- 'id': 'Cinemassacre-521be8ef82b16',
- 'ext': 'mp4',
- 'upload_date': '20131002',
- 'title': 'The Mummy’s Hand (1940)',
- },
- 'params': {
- # m3u8 download
- 'skip_download': True,
- },
- },
- {
- # Youtube embedded video
- 'url': 'http://cinemassacre.com/2006/12/07/chronologically-confused-about-bad-movie-and-video-game-sequel-titles/',
- 'md5': 'ec9838a5520ef5409b3e4e42fcb0a3b9',
- 'info_dict': {
- 'id': 'OEVzPCY2T-g',
- 'ext': 'webm',
- 'title': 'AVGN: Chronologically Confused about Bad Movie and Video Game Sequel Titles',
- 'upload_date': '20061207',
- 'uploader': 'Cinemassacre',
- 'uploader_id': 'JamesNintendoNerd',
- 'description': 'md5:784734696c2b8b7f4b8625cc799e07f6',
- }
- },
- {
- # Youtube embedded video
- 'url': 'http://cinemassacre.com/2006/09/01/mckids/',
- 'md5': '7393c4e0f54602ad110c793eb7a6513a',
- 'info_dict': {
- 'id': 'FnxsNhuikpo',
- 'ext': 'webm',
- 'upload_date': '20060901',
- 'uploader': 'Cinemassacre Extra',
- 'description': 'md5:de9b751efa9e45fbaafd9c8a1123ed53',
- 'uploader_id': 'Cinemassacre',
- 'title': 'AVGN: McKids',
- }
- },
- {
- 'url': 'http://cinemassacre.com/2015/05/25/mario-kart-64-nintendo-64-james-mike-mondays/',
- 'md5': '1376908e49572389e7b06251a53cdd08',
- 'info_dict': {
- 'id': 'Cinemassacre-555779690c440',
- 'ext': 'mp4',
- 'description': 'Let’s Play Mario Kart 64 !! Mario Kart 64 is a classic go-kart racing game released for the Nintendo 64 (N64). Today James & Mike do 4 player Battle Mode with Kyle and Bootsy!',
- 'title': 'Mario Kart 64 (Nintendo 64) James & Mike Mondays',
- 'upload_date': '20150525',
- },
- 'params': {
- # m3u8 download
- 'skip_download': True,
- },
- }
- ]
-
- def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- display_id = mobj.group('display_id')
- video_date = mobj.group('date_y') + mobj.group('date_m') + mobj.group('date_d')
-
- webpage = self._download_webpage(url, display_id)
-
- playerdata_url = self._search_regex(
- [
- ScreenwaveMediaIE.EMBED_PATTERN,
- r'<iframe[^>]+src="(?P<url>(?:https?:)?//(?:[^.]+\.)?youtube\.com/.+?)"',
- ],
- webpage, 'player data URL', default=None, group='url')
- if not playerdata_url:
- raise ExtractorError('Unable to find player data')
-
- video_title = self._html_search_regex(
- r'<title>(?P<title>.+?)\|', webpage, 'title')
- video_description = self._html_search_regex(
- r'<div class="entry-content">(?P<description>.+?)</div>',
- webpage, 'description', flags=re.DOTALL, fatal=False)
- video_thumbnail = self._og_search_thumbnail(webpage)
-
- return {
- '_type': 'url_transparent',
- 'display_id': display_id,
- 'title': video_title,
- 'description': video_description,
- 'upload_date': video_date,
- 'thumbnail': video_thumbnail,
- 'url': playerdata_url,
- }
class CloudyIE(InfoExtractor):
_IE_DESC = 'cloudy.ec and videoraj.ch'
_VALID_URL = r'''(?x)
- https?://(?:www\.)?(?P<host>cloudy\.ec|videoraj\.ch)/
+ https?://(?:www\.)?(?P<host>cloudy\.ec|videoraj\.(?:ch|to))/
(?:v/|embed\.php\?id=)
(?P<id>[A-Za-z0-9]+)
'''
}
},
{
- 'url': 'http://www.videoraj.ch/v/47f399fd8bb60',
+ 'url': 'http://www.videoraj.to/v/47f399fd8bb60',
'md5': '7d0f8799d91efd4eda26587421c3c3b0',
'info_dict': {
'id': '47f399fd8bb60',
+++ /dev/null
-from __future__ import unicode_literals
-
-import json
-import re
-
-from .common import InfoExtractor
-from ..utils import int_or_none
-
-
-class CollegeHumorIE(InfoExtractor):
- _VALID_URL = r'^(?:https?://)?(?:www\.)?collegehumor\.com/(video|embed|e)/(?P<videoid>[0-9]+)/?(?P<shorttitle>.*)$'
-
- _TESTS = [
- {
- 'url': 'http://www.collegehumor.com/video/6902724/comic-con-cosplay-catastrophe',
- 'md5': 'dcc0f5c1c8be98dc33889a191f4c26bd',
- 'info_dict': {
- 'id': '6902724',
- 'ext': 'mp4',
- 'title': 'Comic-Con Cosplay Catastrophe',
- 'description': "Fans get creative this year at San Diego. Too creative. And yes, that's really Joss Whedon.",
- 'age_limit': 13,
- 'duration': 187,
- },
- }, {
- 'url': 'http://www.collegehumor.com/video/3505939/font-conference',
- 'md5': '72fa701d8ef38664a4dbb9e2ab721816',
- 'info_dict': {
- 'id': '3505939',
- 'ext': 'mp4',
- 'title': 'Font Conference',
- 'description': "This video wasn't long enough, so we made it double-spaced.",
- 'age_limit': 10,
- 'duration': 179,
- },
- }, {
- # embedded youtube video
- 'url': 'http://www.collegehumor.com/embed/6950306',
- 'info_dict': {
- 'id': 'Z-bao9fg6Yc',
- 'ext': 'mp4',
- 'title': 'Young Americans Think President John F. Kennedy Died THIS MORNING IN A CAR ACCIDENT!!!',
- 'uploader': 'Mark Dice',
- 'uploader_id': 'MarkDice',
- 'description': 'md5:62c3dab9351fac7bb44b53b69511d87f',
- 'upload_date': '20140127',
- },
- 'params': {
- 'skip_download': True,
- },
- 'add_ie': ['Youtube'],
- },
- ]
-
- def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- video_id = mobj.group('videoid')
-
- jsonUrl = 'http://www.collegehumor.com/moogaloop/video/' + video_id + '.json'
- data = json.loads(self._download_webpage(
- jsonUrl, video_id, 'Downloading info JSON'))
- vdata = data['video']
- if vdata.get('youtubeId') is not None:
- return {
- '_type': 'url',
- 'url': vdata['youtubeId'],
- 'ie_key': 'Youtube',
- }
-
- AGE_LIMITS = {'nc17': 18, 'r': 18, 'pg13': 13, 'pg': 10, 'g': 0}
- rating = vdata.get('rating')
- if rating:
- age_limit = AGE_LIMITS.get(rating.lower())
- else:
- age_limit = None # None = No idea
-
- PREFS = {'high_quality': 2, 'low_quality': 0}
- formats = []
- for format_key in ('mp4', 'webm'):
- for qname, qurl in vdata.get(format_key, {}).items():
- formats.append({
- 'format_id': format_key + '_' + qname,
- 'url': qurl,
- 'format': format_key,
- 'preference': PREFS.get(qname),
- })
- self._sort_formats(formats)
-
- duration = int_or_none(vdata.get('duration'), 1000)
- like_count = int_or_none(vdata.get('likes'))
-
- return {
- 'id': video_id,
- 'title': vdata['title'],
- 'description': vdata.get('description'),
- 'thumbnail': vdata.get('thumbnail'),
- 'formats': formats,
- 'age_limit': age_limit,
- 'duration': duration,
- 'like_count': like_count,
- }
# or: http://www.colbertnation.com/the-colbert-report-collections/422008/festival-of-lights/79524
_VALID_URL = r'''(?x)^(:(?P<shortname>tds|thedailyshow)
|https?://(:www\.)?
- (?P<showname>thedailyshow|thecolbertreport)\.(?:cc\.)?com/
+ (?P<showname>thedailyshow|thecolbertreport|tosh)\.(?:cc\.)?com/
((?:full-)?episodes/(?:[0-9a-z]{6}/)?(?P<episode>.*)|
(?P<clip>
- (?:(?:guests/[^/]+|videos|video-playlists|special-editions|news-team/[^/]+)/[^/]+/(?P<videotitle>[^/?#]+))
+ (?:(?:guests/[^/]+|videos|video-(?:clips|playlists)|special-editions|news-team/[^/]+)/[^/]+/(?P<videotitle>[^/?#]+))
|(the-colbert-report-(videos|collections)/(?P<clipID>[0-9]+)/[^/]*/(?P<cntitle>.*?))
|(watch/(?P<date>[^/]*)/(?P<tdstitle>.*))
)|
}, {
'url': 'http://thedailyshow.cc.com/news-team/michael-che/7wnfel/we-need-to-talk-about-israel',
'only_matching': True,
+ }, {
+ 'url': 'http://tosh.cc.com/video-clips/68g93d/twitter-users-share-summer-plans',
+ 'only_matching': True,
}]
_available_formats = ['3500', '2200', '1700', '1200', '750', '400']
description: Full video description.
uploader: Full name of the video uploader.
license: License name the video is licensed under.
- creator: The main artist who created the video.
+ creator: The creator of the video.
release_date: The date (YYYYMMDD) when the video was released.
timestamp: UNIX timestamp of the moment the video became available.
upload_date: Video upload date (YYYYMMDD).
def _extract_f4m_formats(self, manifest_url, video_id, preference=None, f4m_id=None,
transform_source=lambda s: fix_xml_ampersands(s).strip(),
- fatal=True):
+ fatal=True, m3u8_id=None):
manifest = self._download_xml(
manifest_url, video_id, 'Downloading f4m manifest',
'Unable to download f4m manifest',
return self._parse_f4m_formats(
manifest, manifest_url, video_id, preference=preference, f4m_id=f4m_id,
- transform_source=transform_source, fatal=fatal)
+ transform_source=transform_source, fatal=fatal, m3u8_id=m3u8_id)
def _parse_f4m_formats(self, manifest, manifest_url, video_id, preference=None, f4m_id=None,
transform_source=lambda s: fix_xml_ampersands(s).strip(),
- fatal=True):
+ fatal=True, m3u8_id=None):
+ # currently youtube-dl cannot decode the playerVerificationChallenge as Akamai uses Adobe Alchemy
+ akamai_pv = manifest.find('{http://ns.adobe.com/f4m/1.0}pv-2.0')
+ if akamai_pv is not None and ';' in akamai_pv.text:
+ playerVerificationChallenge = akamai_pv.text.split(';')[0]
+ if playerVerificationChallenge.strip() != '':
+ return []
+
formats = []
manifest_version = '1.0'
media_nodes = manifest.findall('{http://ns.adobe.com/f4m/1.0}media')
'base URL', default=None)
if base_url:
base_url = base_url.strip()
+
+ bootstrap_info = xpath_text(
+ manifest, ['{http://ns.adobe.com/f4m/1.0}bootstrapInfo', '{http://ns.adobe.com/f4m/2.0}bootstrapInfo'],
+ 'bootstrap info', default=None)
+
for i, media_el in enumerate(media_nodes):
- if manifest_version == '2.0':
- media_url = media_el.attrib.get('href') or media_el.attrib.get('url')
+ tbr = int_or_none(media_el.attrib.get('bitrate'))
+ width = int_or_none(media_el.attrib.get('width'))
+ height = int_or_none(media_el.attrib.get('height'))
+ format_id = '-'.join(filter(None, [f4m_id, compat_str(i if tbr is None else tbr)]))
+ # If <bootstrapInfo> is present, the specified f4m is a
+ # stream-level manifest, and only set-level manifests may refer to
+ # external resources. See section 11.4 and section 4 of F4M spec
+ if bootstrap_info is None:
+ media_url = None
+ # @href is introduced in 2.0, see section 11.6 of F4M spec
+ if manifest_version == '2.0':
+ media_url = media_el.attrib.get('href')
+ if media_url is None:
+ media_url = media_el.attrib.get('url')
if not media_url:
continue
manifest_url = (
# since bitrates in parent manifest (this one) and media_url manifest
# may differ leading to inability to resolve the format by requested
# bitrate in f4m downloader
- if determine_ext(manifest_url) == 'f4m':
- formats.extend(self._extract_f4m_formats(
+ ext = determine_ext(manifest_url)
+ if ext == 'f4m':
+ f4m_formats = self._extract_f4m_formats(
manifest_url, video_id, preference=preference, f4m_id=f4m_id,
- transform_source=transform_source, fatal=fatal))
+ transform_source=transform_source, fatal=fatal)
+ # Sometimes stream-level manifest contains single media entry that
+ # does not contain any quality metadata (e.g. http://matchtv.ru/#live-player).
+ # At the same time parent's media entry in set-level manifest may
+ # contain it. We will copy it from parent in such cases.
+ if len(f4m_formats) == 1:
+ f = f4m_formats[0]
+ f.update({
+ 'tbr': f.get('tbr') or tbr,
+ 'width': f.get('width') or width,
+ 'height': f.get('height') or height,
+ 'format_id': f.get('format_id') if not tbr else format_id,
+ })
+ formats.extend(f4m_formats)
+ continue
+ elif ext == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ manifest_url, video_id, 'mp4', preference=preference,
+ m3u8_id=m3u8_id, fatal=fatal))
continue
- tbr = int_or_none(media_el.attrib.get('bitrate'))
formats.append({
- 'format_id': '-'.join(filter(None, [f4m_id, compat_str(i if tbr is None else tbr)])),
+ 'format_id': format_id,
'url': manifest_url,
- 'ext': 'flv',
+ 'ext': 'flv' if bootstrap_info else None,
'tbr': tbr,
- 'width': int_or_none(media_el.attrib.get('width')),
- 'height': int_or_none(media_el.attrib.get('height')),
+ 'width': width,
+ 'height': height,
'preference': preference,
})
return formats
- def _extract_m3u8_formats(self, m3u8_url, video_id, ext=None,
- entry_protocol='m3u8', preference=None,
- m3u8_id=None, note=None, errnote=None,
- fatal=True):
-
- formats = [{
+ def _m3u8_meta_format(self, m3u8_url, ext=None, preference=None, m3u8_id=None):
+ return {
'format_id': '-'.join(filter(None, [m3u8_id, 'meta'])),
'url': m3u8_url,
'ext': ext,
'preference': preference - 1 if preference else -1,
'resolution': 'multiple',
'format_note': 'Quality selection URL',
- }]
+ }
+
+ def _extract_m3u8_formats(self, m3u8_url, video_id, ext=None,
+ entry_protocol='m3u8', preference=None,
+ m3u8_id=None, note=None, errnote=None,
+ fatal=True, live=False):
+
+ formats = [self._m3u8_meta_format(m3u8_url, ext, preference, m3u8_id)]
format_url = lambda u: (
u
format_id = []
if m3u8_id:
format_id.append(m3u8_id)
- last_media_name = last_media.get('NAME') if last_media and last_media.get('TYPE') != 'SUBTITLES' else None
- format_id.append(last_media_name if last_media_name else '%d' % (tbr if tbr else len(formats)))
+ last_media_name = last_media.get('NAME') if last_media and last_media.get('TYPE') not in ('SUBTITLES', 'CLOSED-CAPTIONS') else None
+ # Despite specification does not mention NAME attribute for
+ # EXT-X-STREAM-INF it still sometimes may be present
+ stream_name = last_info.get('NAME') or last_media_name
+ # Bandwidth of live streams may differ over time thus making
+ # format_id unpredictable. So it's better to keep provided
+ # format_id intact.
+ if not live:
+ format_id.append(stream_name if stream_name else '%d' % (tbr if tbr else len(formats)))
f = {
'format_id': '-'.join(format_id),
'url': format_url(line.strip()),
m3u8_count = 0
srcs = []
- videos = smil.findall(self._xpath_ns('.//video', namespace))
- for video in videos:
- src = video.get('src')
+ media = smil.findall(self._xpath_ns('.//video', namespace)) + smil.findall(self._xpath_ns('.//audio', namespace))
+ for medium in media:
+ src = medium.get('src')
if not src or src in srcs:
continue
srcs.append(src)
- bitrate = float_or_none(video.get('system-bitrate') or video.get('systemBitrate'), 1000)
- filesize = int_or_none(video.get('size') or video.get('fileSize'))
- width = int_or_none(video.get('width'))
- height = int_or_none(video.get('height'))
- proto = video.get('proto')
- ext = video.get('ext')
+ bitrate = float_or_none(medium.get('system-bitrate') or medium.get('systemBitrate'), 1000)
+ filesize = int_or_none(medium.get('size') or medium.get('fileSize'))
+ width = int_or_none(medium.get('width'))
+ height = int_or_none(medium.get('height'))
+ proto = medium.get('proto')
+ ext = medium.get('ext')
src_ext = determine_ext(src)
- streamer = video.get('streamer') or base
+ streamer = medium.get('streamer') or base
if proto == 'rtmp' or streamer.startswith('rtmp'):
rtmp_count += 1
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+ ExtractorError,
+ float_or_none,
+ int_or_none,
+ parse_iso8601,
+ qualities,
+)
+
+
+class CoubIE(InfoExtractor):
+ _VALID_URL = r'(?:coub:|https?://(?:coub\.com/(?:view|embed|coubs)/|c-cdn\.coub\.com/fb-player\.swf\?.*\bcoub(?:ID|id)=))(?P<id>[\da-z]+)'
+
+ _TESTS = [{
+ 'url': 'http://coub.com/view/5u5n1',
+ 'info_dict': {
+ 'id': '5u5n1',
+ 'ext': 'mp4',
+ 'title': 'The Matrix Moonwalk',
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ 'duration': 4.6,
+ 'timestamp': 1428527772,
+ 'upload_date': '20150408',
+ 'uploader': 'Артём Лоскутников',
+ 'uploader_id': 'artyom.loskutnikov',
+ 'view_count': int,
+ 'like_count': int,
+ 'repost_count': int,
+ 'comment_count': int,
+ 'age_limit': 0,
+ },
+ }, {
+ 'url': 'http://c-cdn.coub.com/fb-player.swf?bot_type=vk&coubID=7w5a4',
+ 'only_matching': True,
+ }, {
+ 'url': 'coub:5u5n1',
+ 'only_matching': True,
+ }, {
+ # longer video id
+ 'url': 'http://coub.com/view/237d5l5h',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+
+ coub = self._download_json(
+ 'http://coub.com/api/v2/coubs/%s.json' % video_id, video_id)
+
+ if coub.get('error'):
+ raise ExtractorError(
+ '%s said: %s' % (self.IE_NAME, coub['error']), expected=True)
+
+ title = coub['title']
+
+ file_versions = coub['file_versions']
+
+ QUALITIES = ('low', 'med', 'high')
+
+ MOBILE = 'mobile'
+ IPHONE = 'iphone'
+ HTML5 = 'html5'
+
+ SOURCE_PREFERENCE = (MOBILE, IPHONE, HTML5)
+
+ quality_key = qualities(QUALITIES)
+ preference_key = qualities(SOURCE_PREFERENCE)
+
+ formats = []
+
+ for kind, items in file_versions.get(HTML5, {}).items():
+ if kind not in ('video', 'audio'):
+ continue
+ if not isinstance(items, dict):
+ continue
+ for quality, item in items.items():
+ if not isinstance(item, dict):
+ continue
+ item_url = item.get('url')
+ if not item_url:
+ continue
+ formats.append({
+ 'url': item_url,
+ 'format_id': '%s-%s-%s' % (HTML5, kind, quality),
+ 'filesize': int_or_none(item.get('size')),
+ 'vcodec': 'none' if kind == 'audio' else None,
+ 'quality': quality_key(quality),
+ 'preference': preference_key(HTML5),
+ })
+
+ iphone_url = file_versions.get(IPHONE, {}).get('url')
+ if iphone_url:
+ formats.append({
+ 'url': iphone_url,
+ 'format_id': IPHONE,
+ 'preference': preference_key(IPHONE),
+ })
+
+ mobile_url = file_versions.get(MOBILE, {}).get('audio_url')
+ if mobile_url:
+ formats.append({
+ 'url': mobile_url,
+ 'format_id': '%s-audio' % MOBILE,
+ 'preference': preference_key(MOBILE),
+ })
+
+ self._sort_formats(formats)
+
+ thumbnail = coub.get('picture')
+ duration = float_or_none(coub.get('duration'))
+ timestamp = parse_iso8601(coub.get('published_at') or coub.get('created_at'))
+ uploader = coub.get('channel', {}).get('title')
+ uploader_id = coub.get('channel', {}).get('permalink')
+
+ view_count = int_or_none(coub.get('views_count') or coub.get('views_increase_count'))
+ like_count = int_or_none(coub.get('likes_count'))
+ repost_count = int_or_none(coub.get('recoubs_count'))
+ comment_count = int_or_none(coub.get('comments_count'))
+
+ age_restricted = coub.get('age_restricted', coub.get('age_restricted_by_admin'))
+ if age_restricted is not None:
+ age_limit = 18 if age_restricted is True else 0
+ else:
+ age_limit = None
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'thumbnail': thumbnail,
+ 'duration': duration,
+ 'timestamp': timestamp,
+ 'uploader': uploader,
+ 'uploader_id': uploader_id,
+ 'view_count': view_count,
+ 'like_count': like_count,
+ 'repost_count': repost_count,
+ 'comment_count': comment_count,
+ 'age_limit': age_limit,
+ 'formats': formats,
+ }
from .common import InfoExtractor
from ..compat import (
compat_etree_fromstring,
- compat_urllib_parse_unquote,
compat_urllib_parse_urlencode,
compat_urllib_request,
compat_urlparse,
unified_strdate,
urlencode_postdata,
xpath_text,
+ extract_attributes,
)
from ..aes import (
aes_cbc_decrypt,
r'<a[^>]+href="/publisher/[^"]+"[^>]*>([^<]+)</a>', webpage,
'video_uploader', fatal=False)
- playerdata_url = compat_urllib_parse_unquote(self._html_search_regex(r'"config_url":"([^"]+)', webpage, 'playerdata_url'))
- playerdata_req = sanitized_Request(playerdata_url)
- playerdata_req.data = urlencode_postdata({'current_page': webpage_url})
- playerdata_req.add_header('Content-Type', 'application/x-www-form-urlencoded')
- playerdata = self._download_webpage(playerdata_req, video_id, note='Downloading media info')
-
- stream_id = self._search_regex(r'<media_id>([^<]+)', playerdata, 'stream_id')
- video_thumbnail = self._search_regex(r'<episode_image_url>([^<]+)', playerdata, 'thumbnail', fatal=False)
-
+ available_fmts = []
+ for a, fmt in re.findall(r'(<a[^>]+token=["\']showmedia\.([0-9]{3,4})p["\'][^>]+>)', webpage):
+ attrs = extract_attributes(a)
+ href = attrs.get('href')
+ if href and '/freetrial' in href:
+ continue
+ available_fmts.append(fmt)
+ if not available_fmts:
+ for p in (r'token=["\']showmedia\.([0-9]{3,4})p"', r'showmedia\.([0-9]{3,4})p'):
+ available_fmts = re.findall(p, webpage)
+ if available_fmts:
+ break
+ video_encode_ids = []
formats = []
- for fmt in re.findall(r'showmedia\.([0-9]{3,4})p', webpage):
+ for fmt in available_fmts:
stream_quality, stream_format = self._FORMAT_IDS[fmt]
video_format = fmt + 'p'
streamdata_req = sanitized_Request(
'http://www.crunchyroll.com/xml/?req=RpcApiVideoPlayer_GetStandardConfig&media_id=%s&video_format=%s&video_quality=%s'
- % (stream_id, stream_format, stream_quality),
+ % (video_id, stream_format, stream_quality),
compat_urllib_parse_urlencode({'current_page': url}).encode('utf-8'))
streamdata_req.add_header('Content-Type', 'application/x-www-form-urlencoded')
streamdata = self._download_xml(
streamdata_req, video_id,
note='Downloading media info for %s' % video_format)
stream_info = streamdata.find('./{default}preload/stream_info')
+ video_encode_id = xpath_text(stream_info, './video_encode_id')
+ if video_encode_id in video_encode_ids:
+ continue
+ video_encode_ids.append(video_encode_id)
video_url = xpath_text(stream_info, './host')
video_play_path = xpath_text(stream_info, './file')
if not video_url or not video_play_path:
'ext': 'flv',
})
formats.append(format_info)
+ self._sort_formats(formats)
+
+ metadata = self._download_xml(
+ 'http://www.crunchyroll.com/xml', video_id,
+ note='Downloading media info', query={
+ 'req': 'RpcApiVideoPlayer_GetMediaMetadata',
+ 'media_id': video_id,
+ })
subtitles = self.extract_subtitles(video_id, webpage)
'id': video_id,
'title': video_title,
'description': video_description,
- 'thumbnail': video_thumbnail,
+ 'thumbnail': xpath_text(metadata, 'episode_image_url'),
'uploader': video_uploader,
'upload_date': video_upload_date,
+ 'series': xpath_text(metadata, 'series_title'),
+ 'episode': xpath_text(metadata, 'episode_title'),
+ 'episode_number': int_or_none(xpath_text(metadata, 'episode_number')),
'subtitles': subtitles,
'formats': formats,
}
class CWTVIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?cw(?:tv|seed)\.com/shows/(?:[^/]+/){2}\?play=(?P<id>[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})'
+ _VALID_URL = r'https?://(?:www\.)?cw(?:tv|seed)\.com/(?:shows/)?(?:[^/]+/){2}\?.*\bplay=(?P<id>[a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})'
_TESTS = [{
'url': 'http://cwtv.com/shows/arrow/legends-of-yesterday/?play=6b15e985-9345-4f60-baf8-56e96be57c63',
'info_dict': {
# m3u8 download
'skip_download': True,
}
+ }, {
+ 'url': 'http://cwtv.com/thecw/chroniclesofcisco/?play=8adebe35-f447-465f-ab52-e863506ff6d6',
+ 'only_matching': True,
}]
def _real_extract(self, url):
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+ int_or_none,
+ determine_protocol,
+)
+
+
+class DailyMailIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?dailymail\.co\.uk/video/[^/]+/video-(?P<id>[0-9]+)'
+ _TEST = {
+ 'url': 'http://www.dailymail.co.uk/video/sciencetech/video-1288527/Turn-video-impressionist-masterpiece.html',
+ 'md5': '2f639d446394f53f3a33658b518b6615',
+ 'info_dict': {
+ 'id': '1288527',
+ 'ext': 'mp4',
+ 'title': 'Turn any video into an impressionist masterpiece',
+ 'description': 'md5:88ddbcb504367987b2708bb38677c9d2',
+ }
+ }
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ webpage = self._download_webpage(url, video_id)
+ video_data = self._parse_json(self._search_regex(
+ r"data-opts='({.+?})'", webpage, 'video data'), video_id)
+ title = video_data['title']
+ video_sources = self._download_json(video_data.get(
+ 'sources', {}).get('url') or 'http://www.dailymail.co.uk/api/player/%s/video-sources.json' % video_id, video_id)
+
+ formats = []
+ for rendition in video_sources['renditions']:
+ rendition_url = rendition.get('url')
+ if not rendition_url:
+ continue
+ tbr = int_or_none(rendition.get('encodingRate'), 1000)
+ container = rendition.get('videoContainer')
+ is_hls = container == 'M2TS'
+ protocol = 'm3u8_native' if is_hls else determine_protocol({'url': rendition_url})
+ formats.append({
+ 'format_id': ('hls' if is_hls else protocol) + ('-%d' % tbr if tbr else ''),
+ 'url': rendition_url,
+ 'width': int_or_none(rendition.get('frameWidth')),
+ 'height': int_or_none(rendition.get('frameHeight')),
+ 'tbr': tbr,
+ 'vcodec': rendition.get('videoCodec'),
+ 'container': container,
+ 'protocol': protocol,
+ 'ext': 'mp4' if is_hls else None,
+ })
+ self._sort_formats(formats)
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'description': video_data.get('descr'),
+ 'thumbnail': video_data.get('poster') or video_data.get('thumbnail'),
+ 'formats': formats,
+ }
_TEST = {
'url': 'http://tv.dfb.de/video/u-19-em-stimmen-zum-spiel-gegen-russland/11633/',
- # The md5 is different each time
+ 'md5': 'ac0f98a52a330f700b4b3034ad240649',
'info_dict': {
'id': '11633',
'display_id': 'u-19-em-stimmen-zum-spiel-gegen-russland',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'U 19-EM: Stimmen zum Spiel gegen Russland',
'upload_date': '20150714',
},
}
def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- video_id = mobj.group('id')
- display_id = mobj.group('display_id')
+ display_id, video_id = re.match(self._VALID_URL, url).groups()
- webpage = self._download_webpage(url, display_id)
player_info = self._download_xml(
'http://tv.dfb.de/server/hd_video.php?play=%s' % video_id,
display_id)
video_info = player_info.find('video')
-
- f4m_info = self._download_xml(
- self._proto_relative_url(video_info.find('url').text.strip()), display_id)
- token_el = f4m_info.find('token')
- manifest_url = token_el.attrib['url'] + '?' + 'hdnea=' + token_el.attrib['auth'] + '&hdcore=3.2.0'
- formats = self._extract_f4m_formats(manifest_url, display_id)
+ stream_access_url = self._proto_relative_url(video_info.find('url').text.strip())
+
+ formats = []
+ # see http://tv.dfb.de/player/js/ajax.js for the method to extract m3u8 formats
+ for sa_url in (stream_access_url, stream_access_url + '&area=&format=iphone'):
+ stream_access_info = self._download_xml(sa_url, display_id)
+ token_el = stream_access_info.find('token')
+ manifest_url = token_el.attrib['url'] + '?' + 'hdnea=' + token_el.attrib['auth']
+ if '.f4m' in manifest_url:
+ formats.extend(self._extract_f4m_formats(
+ manifest_url + '&hdcore=3.2.0',
+ display_id, f4m_id='hds', fatal=False))
+ else:
+ formats.extend(self._extract_m3u8_formats(
+ manifest_url, display_id, 'mp4',
+ 'm3u8_native', m3u8_id='hls', fatal=False))
self._sort_formats(formats)
return {
'id': video_id,
'display_id': display_id,
'title': video_info.find('title').text,
- 'thumbnail': self._og_search_thumbnail(webpage),
+ 'thumbnail': 'http://tv.dfb.de/images/%s_640x360.jpg' % video_id,
'upload_date': unified_strdate(video_info.find('time_date').text),
'formats': formats,
}
'duration': 156,
'timestamp': 1302032462,
'upload_date': '20110405',
+ 'uploader_id': '103207',
},
'params': {
'skip_download': True, # requires ffmpeg
'upload_date': '20140725',
'timestamp': 1406246400,
'duration': 116,
+ 'uploader_id': '103207',
},
+ 'params': {
+ 'skip_download': True, # requires ffmpeg
+ }
}]
def _real_extract(self, url):
entries = []
for idx, video_info in enumerate(info['playlist']):
- formats = self._extract_m3u8_formats(
- video_info['src'], display_id, 'mp4', 'm3u8_native', m3u8_id='hls',
- note='Download m3u8 information for video %d' % (idx + 1))
- self._sort_formats(formats)
+ subtitles = {}
+ caption_url = video_info.get('captionsUrl')
+ if caption_url:
+ subtitles = {
+ 'en': [{
+ 'url': caption_url,
+ }]
+ }
+
entries.append({
+ '_type': 'url_transparent',
+ 'url': 'http://players.brightcove.net/103207/default_default/index.html?videoId=ref:%s' % video_info['referenceId'],
'id': compat_str(video_info['id']),
- 'formats': formats,
'title': video_info['title'],
'description': video_info.get('description'),
'duration': parse_duration(video_info.get('video_length')),
'thumbnail': video_info.get('thumbnailURL'),
'alt_title': video_info.get('secondary_title'),
'timestamp': parse_iso8601(video_info.get('publishedDate')),
+ 'subtitles': subtitles,
})
return self.playlist_result(entries, display_id, video_title)
'display_id': 'iseven',
'ext': 'flv',
'title': 're:^清晨醒脑！T-ara根本停不下来！ [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
- 'description': 'md5:f34981259a03e980a3c6404190a3ed61',
+ 'description': 're:.*m7show@163\.com.*',
'thumbnail': 're:^https?://.*\.jpg$',
'uploader': '7师傅',
'uploader_id': '431925',
'params': {
'skip_download': True,
},
- 'skip': 'Romm not found',
+ 'skip': 'Room not found',
}, {
'url': 'http://www.douyutv.com/17732',
'info_dict': {
'display_id': '17732',
'ext': 'flv',
'title': 're:^清晨醒脑！T-ara根本停不下来！ [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
- 'description': 'md5:f34981259a03e980a3c6404190a3ed61',
+ 'description': 're:.*m7show@163\.com.*',
'thumbnail': 're:^https?://.*\.jpg$',
'uploader': '7师傅',
'uploader_id': '431925',
room_id = self._html_search_regex(
r'"room_id"\s*:\s*(\d+),', page, 'room id')
- prefix = 'room/%s?aid=android&client_sys=android&time=%d' % (
- room_id, int(time.time()))
-
- auth = hashlib.md5((prefix + '1231').encode('ascii')).hexdigest()
- config = self._download_json(
- 'http://www.douyutv.com/api/v1/%s&auth=%s' % (prefix, auth),
- video_id)
+ config = None
+ # Douyu API sometimes returns error "Unable to load the requested class: eticket_redis_cache"
+ # Retry with different parameters - same parameters cause same errors
+ for i in range(5):
+ prefix = 'room/%s?aid=android&client_sys=android&time=%d' % (
+ room_id, int(time.time()))
+ auth = hashlib.md5((prefix + '1231').encode('ascii')).hexdigest()
+
+ config_page = self._download_webpage(
+ 'http://www.douyutv.com/api/v1/%s&auth=%s' % (prefix, auth),
+ video_id)
+ try:
+ config = self._parse_json(config_page, video_id, fatal=False)
+ except ExtractorError:
+ # Wait some time before retrying to get a different time() value
+ self._sleep(1, video_id, msg_template='%(video_id)s: Error occurs. '
+ 'Waiting for %(timeout)s seconds before retrying')
+ continue
+ else:
+ break
+ if config is None:
+ raise ExtractorError('Unable to fetch API result')
data = config['data']
import time
from .common import InfoExtractor
-from ..utils import int_or_none
+from ..compat import compat_urlparse
+from ..utils import (
+ int_or_none,
+ update_url_query,
+)
class DPlayIE(InfoExtractor):
_VALID_URL = r'https?://(?P<domain>it\.dplay\.com|www\.dplay\.(?:dk|se|no))/[^/]+/(?P<id>[^/?#]+)'
_TESTS = [{
+ # geo restricted, via direct unsigned hls URL
'url': 'http://it.dplay.com/take-me-out/stagione-1-episodio-25/',
'info_dict': {
'id': '1255600',
},
'expected_warnings': ['Unable to download f4m manifest'],
}, {
+ # non geo restricted, via secure api, unsigned download hls URL
'url': 'http://www.dplay.se/nugammalt-77-handelser-som-format-sverige/season-1-svensken-lar-sig-njuta-av-livet/',
'info_dict': {
'id': '3172',
'display_id': 'season-1-svensken-lar-sig-njuta-av-livet',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'Svensken lär sig njuta av livet',
'description': 'md5:d3819c9bccffd0fe458ca42451dd50d8',
'duration': 2650,
'age_limit': 0,
},
}, {
+ # geo restricted, via secure api, unsigned download hls URL
'url': 'http://www.dplay.dk/mig-og-min-mor/season-6-episode-12/',
'info_dict': {
'id': '70816',
'display_id': 'season-6-episode-12',
- 'ext': 'flv',
+ 'ext': 'mp4',
'title': 'Episode 12',
'description': 'md5:9c86e51a93f8a4401fc9641ef9894c90',
'duration': 2563,
'timestamp': 1429696800,
'upload_date': '20150422',
- 'creator': 'Kanal 4',
+ 'creator': 'Kanal 4 (Home)',
'series': 'Mig og min mor',
'season_number': 6,
'episode_number': 12,
'age_limit': 0,
},
}, {
+ # geo restricted, via direct unsigned hls URL
'url': 'http://www.dplay.no/pga-tour/season-1-hoydepunkter-18-21-februar/',
'only_matching': True,
}]
def extract_formats(protocol, manifest_url):
if protocol == 'hls':
- formats.extend(self._extract_m3u8_formats(
+ m3u8_formats = self._extract_m3u8_formats(
manifest_url, video_id, ext='mp4',
- entry_protocol='m3u8_native', m3u8_id=protocol, fatal=False))
+ entry_protocol='m3u8_native', m3u8_id=protocol, fatal=False)
+ # Sometimes final URLs inside m3u8 are unsigned, let's fix this
+ # ourselves
+ query = compat_urlparse.parse_qs(compat_urlparse.urlparse(manifest_url).query)
+ for m3u8_format in m3u8_formats:
+ m3u8_format['url'] = update_url_query(m3u8_format['url'], query)
+ formats.extend(m3u8_formats)
elif protocol == 'hds':
formats.extend(self._extract_f4m_formats(
manifest_url + '&hdcore=3.8.0&plugin=flowplayer-3.8.0.0',
video_id, f4m_id=protocol, fatal=False))
domain_tld = domain.split('.')[-1]
- if domain_tld in ('se', 'dk'):
+ if domain_tld in ('se', 'dk', 'no'):
for protocol in PROTOCOLS:
+ # Providing dsc-geo allows to bypass geo restriction in some cases
self._set_cookie(
'secure.dplay.%s' % domain_tld, 'dsc-geo',
json.dumps({
'Downloading %s stream JSON' % protocol, fatal=False)
if stream and stream.get(protocol):
extract_formats(protocol, stream[protocol])
- else:
+
+ # The last resort is to try direct unsigned hls/hds URLs from info dictionary.
+ # Sometimes this does work even when secure API with dsc-geo has failed (e.g.
+ # http://www.dplay.no/pga-tour/season-1-hoydepunkter-18-21-februar/).
+ if not formats:
for protocol in PROTOCOLS:
if info.get(protocol):
extract_formats(protocol, info[protocol])
self._sort_formats(formats)
+ subtitles = {}
+ for lang in ('se', 'sv', 'da', 'nl', 'no'):
+ for format_id in ('web_vtt', 'vtt', 'srt'):
+ subtitle_url = info.get('subtitles_%s_%s' % (lang, format_id))
+ if subtitle_url:
+ subtitles.setdefault(lang, []).append({'url': subtitle_url})
+
return {
'id': video_id,
'display_id': display_id,
'episode_number': int_or_none(info.get('episode')),
'age_limit': int_or_none(info.get('minimum_age')),
'formats': formats,
+ 'subtitles': subtitles,
}
from __future__ import unicode_literals
from .common import InfoExtractor
-from ..utils import int_or_none
+from ..utils import (
+ int_or_none,
+ unified_strdate,
+)
from ..compat import compat_urlparse
class DWIE(InfoExtractor):
IE_NAME = 'dw'
- _VALID_URL = r'https?://(?:www\.)?dw\.com/(?:[^/]+/)+av-(?P<id>\d+)'
+ _VALID_URL = r'https?://(?:www\.)?dw\.com/(?:[^/]+/)+(?:av|e)-(?P<id>\d+)'
_TESTS = [{
# video
'url': 'http://www.dw.com/en/intelligent-light/av-19112290',
'description': 'md5:bc9ca6e4e063361e21c920c53af12405',
'upload_date': '20160311',
}
+ }, {
+ 'url': 'http://www.dw.com/en/documentaries-welcome-to-the-90s-2016-05-21/e-19220158-9798',
+ 'md5': '56b6214ef463bfb9a3b71aeb886f3cf1',
+ 'info_dict': {
+ 'id': '19274438',
+ 'ext': 'mp4',
+ 'title': 'Welcome to the 90s – Hip Hop',
+ 'description': 'Welcome to the 90s - The Golden Decade of Hip Hop',
+ 'upload_date': '20160521',
+ },
}]
def _real_extract(self, url):
webpage = self._download_webpage(url, media_id)
hidden_inputs = self._hidden_inputs(webpage)
title = hidden_inputs['media_title']
+ media_id = hidden_inputs.get('media_id') or media_id
if hidden_inputs.get('player_type') == 'video' and hidden_inputs.get('stream_file') == '1':
formats = self._extract_smil_formats(
else:
formats = [{'url': hidden_inputs['file_name']}]
+ upload_date = hidden_inputs.get('display_date')
+ if not upload_date:
+ upload_date = self._html_search_regex(
+ r'<span[^>]+class="date">([0-9.]+)\s*\|', webpage,
+ 'upload date', default=None)
+ upload_date = unified_strdate(upload_date)
+
return {
'id': media_id,
'title': title,
'description': self._og_search_description(webpage),
'thumbnail': hidden_inputs.get('preview_image'),
'duration': int_or_none(hidden_inputs.get('file_duration')),
- 'upload_date': hidden_inputs.get('display_date'),
+ 'upload_date': upload_date,
'formats': formats,
}
_TESTS = [{
# http://lenta.ru/news/2015/03/06/navalny/
'url': 'http://lentaru.media.eagleplatform.com/index/player?player=new&record_id=227304&player_template_id=5201',
- 'md5': '881ee8460e1b7735a8be938e2ffb362b',
+ # Not checking MD5 as sometimes the direct HTTP link results in 404 and HLS is used
'info_dict': {
'id': '227304',
'ext': 'mp4',
mobj = re.search('/([^/]+)/index\.m3u8', m3u8_format['url'])
if mobj:
http_format = m3u8_format.copy()
+ video_url = mp4_url.replace(mp4_url_basename, mobj.group(1))
+ if not self._is_valid_url(video_url, video_id):
+ continue
http_format.update({
- 'url': mp4_url.replace(mp4_url_basename, mobj.group(1)),
+ 'url': video_url,
'format_id': m3u8_format['format_id'].replace('hls', 'http'),
'protocol': 'http',
})
class EpornerIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?eporner\.com/hd-porn/(?P<id>\d+)/(?P<display_id>[\w-]+)'
- _TEST = {
+ _VALID_URL = r'https?://(?:www\.)?eporner\.com/hd-porn/(?P<id>\w+)/(?P<display_id>[\w-]+)'
+ _TESTS = [{
'url': 'http://www.eporner.com/hd-porn/95008/Infamous-Tiffany-Teen-Strip-Tease-Video/',
'md5': '39d486f046212d8e1b911c52ab4691f8',
'info_dict': {
'duration': 1838,
'view_count': int,
'age_limit': 18,
- }
- }
+ },
+ }, {
+ # New (May 2016) URL layout
+ 'url': 'http://www.eporner.com/hd-porn/3YRUtzMcWn0/Star-Wars-XXX-Parody/',
+ 'only_matching': True,
+ }]
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
_VALID_URL = r'https?://espn\.go\.com/(?:[^/]+/)*(?P<id>[^/]+)'
_TESTS = [{
'url': 'http://espn.go.com/video/clip?id=10365079',
+ 'md5': '60e5d097a523e767d06479335d1bdc58',
'info_dict': {
'id': 'FkYWtmazr6Ed8xmvILvKLWjd4QvYZpzG',
'ext': 'mp4',
'description': None,
},
'params': {
- # m3u8 download
'skip_download': True,
},
+ 'add_ie': ['OoyalaExternal'],
}, {
# intl video, from http://www.espnfc.us/video/mls-highlights/150/video/2743663/must-see-moments-best-of-the-mls-season
'url': 'http://espn.go.com/video/clip?id=2743663',
+ 'md5': 'f4ac89b59afc7e2d7dbb049523df6768',
'info_dict': {
'id': '50NDFkeTqRHB0nXBOK-RGdSG5YQPuxHg',
'ext': 'mp4',
'title': 'Must-See Moments: Best of the MLS season',
},
'params': {
- # m3u8 download
'skip_download': True,
},
+ 'add_ie': ['OoyalaExternal'],
}, {
'url': 'https://espn.go.com/video/iframe/twitter/?cms=espn&id=10365079',
'only_matching': True,
from .abc import ABCIE
from .abc7news import Abc7NewsIE
+from .abcnews import (
+ AbcNewsIE,
+ AbcNewsVideoIE,
+)
from .academicearth import AcademicEarthCourseIE
from .acast import (
ACastIE,
ArteTVDDCIE,
ArteTVMagazineIE,
ArteTVEmbedIE,
+ ArteTVPlaylistIE,
)
from .atresplayer import AtresPlayerIE
from .atttechchannel import ATTTechChannelIE
from .bild import BildIE
from .bilibili import BiliBiliIE
from .biobiochiletv import BioBioChileTVIE
+from .biqle import BIQLEIE
from .bleacherreport import (
BleacherReportIE,
BleacherReportCMSIE,
CBCPlayerIE,
)
from .cbs import CBSIE
+from .cbslocal import CBSLocalIE
from .cbsinteractive import CBSInteractiveIE
from .cbsnews import (
CBSNewsIE,
ChirbitProfileIE,
)
from .cinchcast import CinchcastIE
-from .cinemassacre import CinemassacreIE
from .cliprs import ClipRsIE
from .clipfish import ClipfishIE
from .cliphunter import CliphunterIE
CNNBlogsIE,
CNNArticleIE,
)
-from .collegehumor import CollegeHumorIE
+from .coub import CoubIE
from .collegerama import CollegeRamaIE
from .comedycentral import ComedyCentralIE, ComedyCentralShowsIE
from .comcarcoff import ComCarCoffIE
from .ctsnews import CtsNewsIE
from .cultureunplugged import CultureUnpluggedIE
from .cwtv import CWTVIE
+from .dailymail import DailyMailIE
from .dailymotion import (
DailymotionIE,
DailymotionPlaylistIE,
from .exfm import ExfmIE
from .expotv import ExpoTVIE
from .extremetube import ExtremeTubeIE
+from .eyedotv import EyedoTVIE
from .facebook import FacebookIE
from .faz import FazIE
from .fc2 import FC2IE
from .flickr import FlickrIE
from .folketinget import FolketingetIE
from .footyroom import FootyRoomIE
+from .formula1 import Formula1IE
from .fourtube import FourTubeIE
from .fox import FOXIE
from .foxgay import FoxgayIE
)
from .la7 import LA7IE
from .laola1tv import Laola1TvIE
+from .learnr import LearnrIE
from .lecture2go import Lecture2GoIE
from .lemonde import LemondeIE
from .leeco import (
LePlaylistIE,
LetvCloudIE,
)
+from .libraryofcongress import LibraryOfCongressIE
from .libsyn import LibsynIE
from .lifenews import (
LifeNewsIE,
LimelightChannelIE,
LimelightChannelListIE,
)
+from .litv import LiTVIE
from .liveleak import LiveLeakIE
from .livestream import (
LivestreamIE,
LivestreamShortenerIE,
)
from .lnkgo import LnkGoIE
+from .localnews8 import LocalNews8IE
from .lovehomeporn import LoveHomePornIE
from .lrt import LRTIE
from .lynda import (
from .mailru import MailRuIE
from .makerschannel import MakersChannelIE
from .makertv import MakerTVIE
-from .malemotion import MalemotionIE
from .matchtv import MatchTVIE
from .mdr import MDRIE
from .metacafe import MetacafeIE
from .metacritic import MetacriticIE
from .mgoon import MgoonIE
from .mgtv import MGTVIE
+from .microsoftvirtualacademy import (
+ MicrosoftVirtualAcademyIE,
+ MicrosoftVirtualAcademyCourseIE,
+)
from .minhateca import MinhatecaIE
from .ministrygrid import MinistryGridIE
from .minoto import MinotoIE
)
from .muenchentv import MuenchenTVIE
from .musicplayon import MusicPlayOnIE
-from .muzu import MuzuTVIE
-from .mwave import MwaveIE
+from .mwave import MwaveIE, MwaveMeetGreetIE
from .myspace import MySpaceIE, MySpaceAlbumIE
from .myspass import MySpassIE
from .myvi import MyviIE
from .patreon import PatreonIE
from .pbs import PBSIE
from .people import PeopleIE
-from .periscope import PeriscopeIE
+from .periscope import (
+ PeriscopeIE,
+ PeriscopeUserIE,
+)
from .philharmoniedeparis import PhilharmonieDeParisIE
from .phoenix import PhoenixIE
from .photobucket import PhotobucketIE
QQMusicPlaylistIE,
)
from .r7 import R7IE
+from .radiocanada import (
+ RadioCanadaIE,
+ RadioCanadaAudioVideoIE,
+)
from .radiode import RadioDeIE
from .radiojavan import RadioJavanIE
from .radiobremen import RadioBremenIE
from .redtube import RedTubeIE
from .regiotv import RegioTVIE
from .restudy import RestudyIE
+from .reuters import ReutersIE
from .reverbnation import ReverbNationIE
-from .revision3 import Revision3IE
+from .revision3 import (
+ Revision3EmbedIE,
+ Revision3IE,
+)
from .rice import RICEIE
from .ringtv import RingTVIE
from .ro220 import Ro220IE
from .screencastomatic import ScreencastOMaticIE
from .screenjunkies import ScreenJunkiesIE
from .screenwavemedia import ScreenwaveMediaIE, TeamFourIE
+from .seeker import SeekerIE
from .senateisvp import SenateISVPIE
+from .sendtonews import SendtoNewsIE
from .servingsys import ServingSysIE
from .sexu import SexuIE
-from .sexykarma import SexyKarmaIE
from .shahid import ShahidIE
from .shared import SharedIE
from .sharesix import ShareSixIE
SmotriUserIE,
SmotriBroadcastIE,
)
-from .snagfilms import (
- SnagFilmsIE,
- SnagFilmsEmbedIE,
-)
from .snotr import SnotrIE
from .sohu import SohuIE
from .soundcloud import (
from .swrmediathek import SWRMediathekIE
from .syfy import SyfyIE
from .sztvhu import SztvHuIE
-from .tagesschau import TagesschauIE
+from .tagesschau import (
+ TagesschauPlayerIE,
+ TagesschauIE,
+)
from .tapely import TapelyIE
from .tass import TassIE
from .tdslifeway import TDSLifewayIE
from .thestar import TheStarIE
from .thisamericanlife import ThisAmericanLifeIE
from .thisav import ThisAVIE
+from .threeqsdn import ThreeQSDNIE
from .tinypic import TinyPicIE
from .tlc import TlcDeIE
from .tmz import (
)
from .tvigle import TvigleIE
from .tvland import TVLandIE
-from .tvp import TvpIE, TvpSeriesIE
+from .tvp import (
+ TVPIE,
+ TVPSeriesIE,
+)
from .tvplay import TVPlayIE
from .tweakers import TweakersIE
from .twentyfourvideo import TwentyFourVideoIE
TwitchVodIE,
TwitchProfileIE,
TwitchPastBroadcastsIE,
- TwitchBookmarksIE,
TwitchStreamIE,
)
from .twitter import (
from .urort import UrortIE
from .usatoday import USATodayIE
from .ustream import UstreamIE, UstreamChannelIE
-from .ustudio import UstudioIE
+from .ustudio import (
+ UstudioIE,
+ UstudioEmbedIE,
+)
from .varzesh3 import Varzesh3IE
from .vbox7 import Vbox7IE
from .veehd import VeeHDIE
from .veoh import VeohIE
from .vessel import VesselIE
from .vesti import VestiIE
-from .vevo import VevoIE
+from .vevo import (
+ VevoIE,
+ VevoPlaylistIE,
+)
from .vgtv import (
BTArticleIE,
BTVestlendingenIE,
)
from .videopremium import VideoPremiumIE
from .videott import VideoTtIE
+from .vidio import VidioIE
from .vidme import (
VidmeIE,
VidmeUserIE,
)
from .vidzi import VidziIE
from .vier import VierIE, VierVideosIE
+from .viewlift import (
+ ViewLiftIE,
+ ViewLiftEmbedIE,
+)
from .viewster import ViewsterIE
from .viidea import ViideaIE
from .vimeo import (
from .vuclip import VuClipIE
from .vulture import VultureIE
from .walla import WallaIE
-from .washingtonpost import WashingtonPostIE
+from .washingtonpost import (
+ WashingtonPostIE,
+ WashingtonPostArticleIE,
+)
from .wat import WatIE
+from .watchindianporn import WatchIndianPornIE
from .wdr import (
WDRIE,
WDRMobileIE,
XHamsterIE,
XHamsterEmbedIE,
)
+from .xiami import (
+ XiamiSongIE,
+ XiamiAlbumIE,
+ XiamiArtistIE,
+ XiamiCollectionIE
+)
from .xminus import XMinusIE
from .xnxx import XNXXIE
from .xstream import XstreamIE
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+ xpath_text,
+ parse_duration,
+ ExtractorError,
+)
+
+
+class EyedoTVIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?eyedo\.tv/[^/]+/(?:#!/)?Live/Detail/(?P<id>[0-9]+)'
+ _TEST = {
+ 'url': 'https://www.eyedo.tv/en-US/#!/Live/Detail/16301',
+ 'md5': 'ba14f17995cdfc20c36ba40e21bf73f7',
+ 'info_dict': {
+ 'id': '16301',
+ 'ext': 'mp4',
+ 'title': 'Journée du conseil scientifique de l\'Afnic 2015',
+ 'description': 'md5:4abe07293b2f73efc6e1c37028d58c98',
+ 'uploader': 'Afnic Live',
+ 'uploader_id': '8023',
+ }
+ }
+ _ROOT_URL = 'http://live.eyedo.net:1935/'
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ video_data = self._download_xml('http://eyedo.tv/api/live/GetLive/%s' % video_id, video_id)
+
+ def _add_ns(path):
+ return self._xpath_ns(path, 'http://schemas.datacontract.org/2004/07/EyeDo.Core.Implementation.Web.ViewModels.Api')
+
+ title = xpath_text(video_data, _add_ns('Titre'), 'title', True)
+ state_live_code = xpath_text(video_data, _add_ns('StateLiveCode'), 'title', True)
+ if state_live_code == 'avenir':
+ raise ExtractorError(
+ '%s said: We\'re sorry, but this video is not yet available.' % self.IE_NAME,
+ expected=True)
+
+ is_live = state_live_code == 'live'
+ m3u8_url = None
+ # http://eyedo.tv/Content/Html5/Scripts/html5view.js
+ if is_live:
+ if xpath_text(video_data, 'Cdn') == 'true':
+ m3u8_url = 'http://rrr.sz.xlcdn.com/?account=eyedo&file=A%s&type=live&service=wowza&protocol=http&output=playlist.m3u8' % video_id
+ else:
+ m3u8_url = self._ROOT_URL + 'w/%s/eyedo_720p/playlist.m3u8' % video_id
+ else:
+ m3u8_url = self._ROOT_URL + 'replay-w/%s/mp4:%s.mp4/playlist.m3u8' % (video_id, video_id)
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'formats': self._extract_m3u8_formats(
+ m3u8_url, video_id, 'mp4', 'm3u8' if is_live else 'm3u8_native'),
+ 'description': xpath_text(video_data, _add_ns('Description')),
+ 'duration': parse_duration(xpath_text(video_data, _add_ns('Duration'))),
+ 'uploader': xpath_text(video_data, _add_ns('Createur')),
+ 'uploader_id': xpath_text(video_data, _add_ns('CreateurId')),
+ 'chapter': xpath_text(video_data, _add_ns('ChapitreTitre')),
+ 'chapter_id': xpath_text(video_data, _add_ns('ChapitreId')),
+ }
# coding: utf-8
from __future__ import unicode_literals
-import re
-
from .common import InfoExtractor
+from ..compat import compat_urlparse
class FczenitIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?fc-zenit\.ru/video/gl(?P<id>[0-9]+)'
+ _VALID_URL = r'https?://(?:www\.)?fc-zenit\.ru/video/(?P<id>[0-9]+)'
_TEST = {
- 'url': 'http://fc-zenit.ru/video/gl6785/',
- 'md5': '458bacc24549173fe5a5aa29174a5606',
+ 'url': 'http://fc-zenit.ru/video/41044/',
+ 'md5': '0e3fab421b455e970fa1aa3891e57df0',
'info_dict': {
- 'id': '6785',
+ 'id': '41044',
'ext': 'mp4',
- 'title': '«Зенит-ТВ»: как Олег Шатов играл против «Урала»',
+ 'title': 'Так пишется история: казанский разгром ЦСКА на «Зенит-ТВ»',
},
}
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
- video_title = self._html_search_regex(r'<div class=\"photoalbum__title\">([^<]+)', webpage, 'title')
+ video_title = self._html_search_regex(
+ r'<[^>]+class=\"photoalbum__title\">([^<]+)', webpage, 'title')
+
+ video_items = self._parse_json(self._search_regex(
+ r'arrPath\s*=\s*JSON\.parse\(\'(.+)\'\)', webpage, 'video items'),
+ video_id)
- bitrates_raw = self._html_search_regex(r'bitrates:.*\n(.*)\]', webpage, 'video URL')
- bitrates = re.findall(r'url:.?\'(.+?)\'.*?bitrate:.?([0-9]{3}?)', bitrates_raw)
+ def merge_dicts(*dicts):
+ ret = {}
+ for a_dict in dicts:
+ ret.update(a_dict)
+ return ret
formats = [{
- 'url': furl,
- 'tbr': tbr,
- } for furl, tbr in bitrates]
+ 'url': compat_urlparse.urljoin(url, video_url),
+ 'tbr': int(tbr),
+ } for tbr, video_url in merge_dicts(*video_items).items()]
self._sort_formats(formats)
'upload_date': '20110423',
'uploader_id': '10922353@N03',
'uploader': 'Forest Wander',
+ 'uploader_url': 'https://www.flickr.com/photos/forestwander-nature-pictures/',
'comment_count': int,
'view_count': int,
'tags': list,
+ 'license': 'Attribution-ShareAlike',
}
}
-
_API_BASE_URL = 'https://api.flickr.com/services/rest?'
+ # https://help.yahoo.com/kb/flickr/SLN25525.html
+ _LICENSES = {
+ '0': 'All Rights Reserved',
+ '1': 'Attribution-NonCommercial-ShareAlike',
+ '2': 'Attribution-NonCommercial',
+ '3': 'Attribution-NonCommercial-NoDerivs',
+ '4': 'Attribution',
+ '5': 'Attribution-ShareAlike',
+ '6': 'Attribution-NoDerivs',
+ '7': 'No known copyright restrictions',
+ '8': 'United States government work',
+ '9': 'Public Domain Dedication (CC0)',
+ '10': 'Public Domain Work',
+ }
def _call_api(self, method, video_id, api_key, note, secret=None):
query = {
self._sort_formats(formats)
owner = video_info.get('owner', {})
+ uploader_id = owner.get('nsid')
+ uploader_path = owner.get('path_alias') or uploader_id
+ uploader_url = 'https://www.flickr.com/photos/%s/' % uploader_path if uploader_path else None
return {
'id': video_id,
'formats': formats,
'timestamp': int_or_none(video_info.get('dateuploaded')),
'duration': int_or_none(video_info.get('video', {}).get('duration')),
- 'uploader_id': owner.get('nsid'),
+ 'uploader_id': uploader_id,
'uploader': owner.get('realname'),
+ 'uploader_url': uploader_url,
'comment_count': int_or_none(video_info.get('comments', {}).get('_content')),
'view_count': int_or_none(video_info.get('views')),
- 'tags': [tag.get('_content') for tag in video_info.get('tags', {}).get('tag', [])]
+ 'tags': [tag.get('_content') for tag in video_info.get('tags', {}).get('tag', [])],
+ 'license': self._LICENSES.get(video_info.get('license')),
}
else:
raise ExtractorError('not a video', expected=True)
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class Formula1IE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?formula1\.com/content/fom-website/en/video/\d{4}/\d{1,2}/(?P<id>.+?)\.html'
+ _TEST = {
+ 'url': 'http://www.formula1.com/content/fom-website/en/video/2016/5/Race_highlights_-_Spain_2016.html',
+ 'md5': '8c79e54be72078b26b89e0e111c0502b',
+ 'info_dict': {
+ 'id': 'JvYXJpMzE6pArfHWm5ARp5AiUmD-gibV',
+ 'ext': 'flv',
+ 'title': 'Race highlights - Spain 2016',
+ },
+ 'add_ie': ['Ooyala'],
+ }
+
+ def _real_extract(self, url):
+ display_id = self._match_id(url)
+ webpage = self._download_webpage(url, display_id)
+ ooyala_embed_code = self._search_regex(
+ r'data-videoid="([^"]+)"', webpage, 'ooyala embed code')
+ return self.url_result(
+ 'ooyala:%s' % ooyala_embed_code, 'Ooyala', ooyala_embed_code)
from __future__ import unicode_literals
from .common import InfoExtractor
+from ..compat import (
+ compat_HTTPError,
+ compat_urllib_parse_unquote_plus,
+)
from ..utils import (
clean_html,
determine_ext,
'description': 'md5:1769f43cd5fc130ace8fd87232207892',
'thumbnail': 're:https?://.*\.jpg',
},
+ 'skip': 'Access without user interaction is forbidden by CloudFlare, and video removed',
}, {
'url': 'http://www.funimation.com/shows/hacksign/videos/official/role-play',
'info_dict': {
'description': 'md5:b602bdc15eef4c9bbb201bb6e6a4a2dd',
'thumbnail': 're:https?://.*\.jpg',
},
+ 'skip': 'Access without user interaction is forbidden by CloudFlare',
}, {
'url': 'http://www.funimation.com/shows/attack-on-titan-junior-high/videos/promotional/broadcast-dub-preview',
'info_dict': {
'description': 'md5:f8ec49c0aff702a7832cd81b8a44f803',
'thumbnail': 're:https?://.*\.(?:jpg|png)',
},
+ 'skip': 'Access without user interaction is forbidden by CloudFlare',
}]
+ _LOGIN_URL = 'http://www.funimation.com/login'
+
+ def _download_webpage(self, *args, **kwargs):
+ try:
+ return super(FunimationIE, self)._download_webpage(*args, **kwargs)
+ except ExtractorError as ee:
+ if isinstance(ee.cause, compat_HTTPError) and ee.cause.code == 403:
+ response = ee.cause.read()
+ if b'>Please complete the security check to access<' in response:
+ raise ExtractorError(
+ 'Access to funimation.com is blocked by CloudFlare. '
+ 'Please browse to http://www.funimation.com/, solve '
+ 'the reCAPTCHA, export browser cookies to a text file,'
+ ' and then try again with --cookies YOUR_COOKIE_FILE.',
+ expected=True)
+ raise
+
+ def _extract_cloudflare_session_ua(self, url):
+ ci_session_cookie = self._get_cookies(url).get('ci_session')
+ if ci_session_cookie:
+ ci_session = compat_urllib_parse_unquote_plus(ci_session_cookie.value)
+ # ci_session is a string serialized by PHP function serialize()
+ # This case is simple enough to use regular expressions only
+ return self._search_regex(
+ r'"user_agent";s:\d+:"([^"]+)"', ci_session, 'user agent',
+ default=None)
+
def _login(self):
(username, password) = self._get_login_info()
if username is None:
'email_field': username,
'password_field': password,
})
- login_request = sanitized_Request('http://www.funimation.com/login', data, headers={
- 'User-Agent': 'Mozilla/5.0 (Windows NT 5.2; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0',
+ user_agent = self._extract_cloudflare_session_ua(self._LOGIN_URL)
+ if not user_agent:
+ user_agent = 'Mozilla/5.0 (Windows NT 5.2; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0'
+ login_request = sanitized_Request(self._LOGIN_URL, data, headers={
+ 'User-Agent': user_agent,
'Content-Type': 'application/x-www-form-urlencoded'
})
login_page = self._download_webpage(
('mobile', 'Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36'),
)
+ user_agent = self._extract_cloudflare_session_ua(url)
+ if user_agent:
+ USER_AGENTS = ((None, user_agent),)
+
for kind, user_agent in USER_AGENTS:
request = sanitized_Request(url)
request.add_header('User-Agent', user_agent)
webpage = self._download_webpage(
- request, display_id, 'Downloading %s webpage' % kind)
+ request, display_id,
+ 'Downloading %s webpage' % kind if kind else 'Downloading webpage')
playlist = self._parse_json(
self._search_regex(
from .vimeo import VimeoIE
from .dailymotion import DailymotionCloudIE
from .onionstudios import OnionStudiosIE
-from .snagfilms import SnagFilmsEmbedIE
+from .viewlift import ViewLiftEmbedIE
from .screenwavemedia import ScreenwaveMediaIE
from .mtv import MTVServicesEmbeddedIE
from .pladform import PladformIE
from .digiteka import DigitekaIE
from .instagram import InstagramIE
from .liveleak import LiveLeakIE
+from .threeqsdn import ThreeQSDNIE
+from .theplatform import ThePlatformIE
class GenericIE(InfoExtractor):
'ext': 'mp4',
'title': 'car-20120827-manifest',
'formats': 'mincount:9',
+ 'upload_date': '20130904',
},
'params': {
'format': 'bestvideo',
'id': 'k2mm4bCdJ6CQ2i7c8o2',
'ext': 'mp4',
'title': 'Le Zap de Spi0n n°216 - Zapping du Web',
+ 'description': 'md5:faf028e48a461b8b7fad38f1e104b119',
'uploader': 'Spi0n',
+ 'uploader_id': 'xgditw',
+ 'upload_date': '20140425',
+ 'timestamp': 1398441542,
},
'add_ie': ['Dailymotion'],
},
},
# Wistia embed
{
- 'url': 'http://education-portal.com/academy/lesson/north-american-exploration-failed-colonies-of-spain-france-england.html#lesson',
- 'md5': '8788b683c777a5cf25621eaf286d0c23',
+ 'url': 'http://study.com/academy/lesson/north-american-exploration-failed-colonies-of-spain-france-england.html#lesson',
+ 'md5': '1953f3a698ab51cfc948ed3992a0b7ff',
'info_dict': {
- 'id': '1cfaf6b7ea',
+ 'id': '6e2wtrbdaf',
'ext': 'mov',
- 'title': 'md5:51364a8d3d009997ba99656004b5e20d',
- 'duration': 643.0,
- 'filesize': 182808282,
- 'uploader': 'education-portal.com',
+ 'title': 'paywall_north-american-exploration-failed-colonies-of-spain-france-england',
+ 'description': 'a Paywall Videos video from Remilon',
+ 'duration': 644.072,
+ 'uploader': 'study.com',
+ 'timestamp': 1459678540,
+ 'upload_date': '20160403',
+ 'filesize': 24687186,
},
},
{
'info_dict': {
'id': 'uxjb0lwrcz',
'ext': 'mp4',
- 'title': 'Conversation about Hexagonal Rails Part 1 - ThoughtWorks',
+ 'title': 'Conversation about Hexagonal Rails Part 1',
+ 'description': 'a Martin Fowler video from ThoughtWorks',
'duration': 1715.0,
'uploader': 'thoughtworks.wistia.com',
+ 'timestamp': 1401832161,
+ 'upload_date': '20140603',
},
},
+ # Wistia standard embed (async)
+ {
+ 'url': 'https://www.getdrip.com/university/brennan-dunn-drip-workshop/',
+ 'info_dict': {
+ 'id': '807fafadvk',
+ 'ext': 'mp4',
+ 'title': 'Drip Brennan Dunn Workshop',
+ 'description': 'a JV Webinars video from getdrip-1',
+ 'duration': 4986.95,
+ 'timestamp': 1463607249,
+ 'upload_date': '20160518',
+ },
+ 'params': {
+ 'skip_download': True,
+ }
+ },
# Soundcloud embed
{
'url': 'http://nakedsecurity.sophos.com/2014/10/29/sscc-171-are-you-sure-that-1234-is-a-bad-password-podcast/',
'title': 'Rosetta #CometLanding webcast HL 10',
}
},
+ # Another Livestream embed, without 'new.' in URL
+ {
+ 'url': 'https://www.freespeech.org/',
+ 'info_dict': {
+ 'id': '123537347',
+ 'ext': 'mp4',
+ 'title': 're:^FSTV [0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}$',
+ },
+ 'params': {
+ # Live stream
+ 'skip_download': True,
+ },
+ },
# LazyYT
{
'url': 'http://discourse.ubuntu.com/t/unity-8-desktop-mode-windows-on-mir/1986',
'title': 'EP3S5 - Bon Appétit - Baqueira Mi Corazon !',
}
},
- # Kaltura embed
- {
- 'url': 'http://www.monumentalnetwork.com/videos/john-carlson-postgame-2-25-15',
- 'info_dict': {
- 'id': '1_eergr3h1',
- 'ext': 'mp4',
- 'upload_date': '20150226',
- 'uploader_id': 'MonumentalSports-Kaltura@perfectsensedigital.com',
- 'timestamp': int,
- 'title': 'John Carlson Postgame 2/25/15',
- },
- },
# Kaltura embed (different embed code)
{
'url': 'http://www.premierchristianradio.com/Shows/Saturday/Unbelievable/Conference-Videos/Os-Guinness-Is-It-Fools-Talk-Unbelievable-Conference-2014',
'uploader_id': 'echojecka',
},
},
+ # Kaltura embed with single quotes
+ {
+ 'url': 'http://fod.infobase.com/p_ViewPlaylist.aspx?AssignmentID=NUN8ZY',
+ 'info_dict': {
+ 'id': '0_izeg5utt',
+ 'ext': 'mp4',
+ 'title': '35871',
+ 'timestamp': 1355743100,
+ 'upload_date': '20121217',
+ 'uploader_id': 'batchUser',
+ },
+ 'add_ie': ['Kaltura'],
+ },
# Eagle.Platform embed (generic URL)
{
'url': 'http://lenta.ru/news/2015/03/06/navalny/',
+ # Not checking MD5 as sometimes the direct HTTP link results in 404 and HLS is used
'info_dict': {
'id': '227304',
'ext': 'mp4',
# ClipYou (Eagle.Platform) embed (custom URL)
{
'url': 'http://muz-tv.ru/play/7129/',
+ # Not checking MD5 as sometimes the direct HTTP link results in 404 and HLS is used
'info_dict': {
'id': '12820',
'ext': 'mp4',
'ext': 'flv',
'title': "PFT Live: New leader in the 'new-look' defense",
'description': 'md5:65a19b4bbfb3b0c0c5768bed1dfad74e',
+ 'uploader': 'NBCU-SPORTS',
+ 'upload_date': '20140107',
+ 'timestamp': 1389118457,
},
},
# UDN embed
{
- 'url': 'http://www.udn.com/news/story/7314/822787',
+ 'url': 'https://video.udn.com/news/300346',
'md5': 'fd2060e988c326991037b9aff9df21a6',
'info_dict': {
'id': '300346',
'ext': 'mp4',
'title': '中一中男師變性 全校師生力挺',
'thumbnail': 're:^https?://.*\.jpg$',
- }
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
},
# Ooyala embed
{
'title': 'SN Presents: Russell Martin, World Citizen',
'description': 'To understand why he was the Toronto Blue Jays’ top off-season priority is to appreciate his background and upbringing in Montreal, where he first developed his baseball skills. Written and narrated by Stephen Brunt.',
'uploader': 'Rogers Sportsnet',
+ 'uploader_id': '1704050871',
+ 'upload_date': '20150525',
+ 'timestamp': 1432570283,
},
},
# Dailymotion Cloud video
'title': 'The Cardinal Pell Interview',
'description': 'Sky News Contributor Andrew Bolt interviews George Pell in Rome, following the Cardinal\'s evidence before the Royal Commission into Child Abuse. ',
'uploader': 'GlobeCast Australia - GlobeStream',
+ 'uploader_id': '2733773828001',
+ 'upload_date': '20160304',
+ 'timestamp': 1457083087,
},
'params': {
# m3u8 downloads
'uploader': 'Lake8737',
}
},
+ # Duplicated embedded video URLs
+ {
+ 'url': 'http://www.hudl.com/athlete/2538180/highlights/149298443',
+ 'info_dict': {
+ 'id': '149298443_480_16c25b74_2',
+ 'ext': 'mp4',
+ 'title': 'vs. Blue Orange Spring Game',
+ 'uploader': 'www.hudl.com',
+ },
+ },
]
def report_following_redirect(self, new_url):
# Site Name | Video Title
# Video Title - Tagline | Site Name
# and so on and so forth; it's just not practical
- video_title = self._html_search_regex(
+ video_title = self._og_search_title(
+ webpage, default=None) or self._html_search_regex(
r'(?s)<title>(.*?)</title>', webpage, 'video title',
default='video')
video_uploader = self._search_regex(
r'^(?:https?://)?([^/]*)/.*', url, 'video uploader')
+ video_description = self._og_search_description(webpage, default=None)
+ video_thumbnail = self._og_search_thumbnail(webpage, default=None)
+
# Helper method
def _playlist_from_matches(matches, getter=None, ie=None):
urlrs = orderedSet(
if bc_urls:
return _playlist_from_matches(bc_urls, ie='BrightcoveNew')
+ # Look for ThePlatform embeds
+ tp_urls = ThePlatformIE._extract_urls(webpage)
+ if tp_urls:
+ return _playlist_from_matches(tp_urls, ie='ThePlatform')
+
# Look for embedded rtl.nl player
matches = re.findall(
r'<iframe[^>]+?src="((?:https?:)?//(?:www\.)?rtl\.nl/system/videoplayer/[^"]+(?:video_)?embed[^"]+)"',
'url': embed_url,
'ie_key': 'Wistia',
'uploader': video_uploader,
- 'title': video_title,
- 'id': video_id,
}
match = re.search(r'(?:id=["\']wistia_|data-wistia-?id=["\']|Wistia\.embed\(["\'])(?P<id>[^"\']+)', webpage)
if match:
return {
'_type': 'url_transparent',
- 'url': 'http://fast.wistia.net/embed/iframe/{0:}'.format(match.group('id')),
+ 'url': 'wistia:%s' % match.group('id'),
'ie_key': 'Wistia',
'uploader': video_uploader,
- 'title': video_title,
- 'id': match.group('id')
}
+ match = re.search(
+ r'''(?sx)
+ <script[^>]+src=(["'])(?:https?:)?//fast\.wistia\.com/assets/external/E-v1\.js\1[^>]*>.*?
+ <div[^>]+class=(["']).*?\bwistia_async_(?P<id>[a-z0-9]+)\b.*?\2
+ ''', webpage)
+ if match:
+ return self.url_result(self._proto_relative_url(
+ 'wistia:%s' % match.group('id')), 'Wistia')
+
# Look for SVT player
svt_url = SVTIE._extract_url(webpage)
if svt_url:
return self.url_result(self._proto_relative_url(mobj.group('url'), scheme='http:'), 'CondeNast')
mobj = re.search(
- r'<iframe[^>]+src="(?P<url>https?://new\.livestream\.com/[^"]+/player[^"]+)"',
+ r'<iframe[^>]+src="(?P<url>https?://(?:new\.)?livestream\.com/[^"]+/player[^"]+)"',
webpage)
if mobj is not None:
return self.url_result(mobj.group('url'), 'Livestream')
return self.url_result(mobj.group('url'), 'Zapiks')
# Look for Kaltura embeds
- mobj = (re.search(r"(?s)kWidget\.(?:thumb)?[Ee]mbed\(\{.*?'wid'\s*:\s*'_?(?P<partner_id>[^']+)',.*?'entry_?[Ii]d'\s*:\s*'(?P<id>[^']+)',", webpage) or
+ mobj = (re.search(r"(?s)kWidget\.(?:thumb)?[Ee]mbed\(\{.*?(?P<q1>['\"])wid(?P=q1)\s*:\s*(?P<q2>['\"])_?(?P<partner_id>[^'\"]+)(?P=q2),.*?(?P<q3>['\"])entry_?[Ii]d(?P=q3)\s*:\s*(?P<q4>['\"])(?P<id>[^'\"]+)(?P=q4),", webpage) or
re.search(r'(?s)(?P<q1>["\'])(?:https?:)?//cdnapi(?:sec)?\.kaltura\.com/.*?(?:p|partner_id)/(?P<partner_id>\d+).*?(?P=q1).*?entry_?[Ii]d\s*:\s*(?P<q2>["\'])(?P<id>.+?)(?P=q2)', webpage))
if mobj is not None:
return self.url_result(smuggle_url(
if onionstudios_url:
return self.url_result(onionstudios_url)
- # Look for SnagFilms embeds
- snagfilms_url = SnagFilmsEmbedIE._extract_url(webpage)
- if snagfilms_url:
- return self.url_result(snagfilms_url)
+ # Look for ViewLift embeds
+ viewlift_url = ViewLiftEmbedIE._extract_url(webpage)
+ if viewlift_url:
+ return self.url_result(viewlift_url)
# Look for JWPlatform embeds
jwplatform_url = JWPlatformIE._extract_url(webpage)
if liveleak_url:
return self.url_result(liveleak_url, 'LiveLeak')
+ # Look for 3Q SDN embeds
+ threeqsdn_url = ThreeQSDNIE._extract_url(webpage)
+ if threeqsdn_url:
+ return {
+ '_type': 'url_transparent',
+ 'ie_key': ThreeQSDNIE.ie_key(),
+ 'url': self._proto_relative_url(threeqsdn_url),
+ 'title': video_title,
+ 'description': video_description,
+ 'thumbnail': video_thumbnail,
+ 'uploader': video_uploader,
+ }
+
def check_video(vurl):
if YoutubeIE.suitable(vurl):
return True
raise UnsupportedError(url)
entries = []
- for video_url in found:
+ for video_url in orderedSet(found):
+ video_url = unescapeHTML(video_url)
video_url = video_url.replace('\\/', '/')
video_url = compat_urlparse.urljoin(url, video_url)
video_id = compat_urllib_parse_unquote(os.path.basename(video_url))
class GrouponIE(InfoExtractor):
- _VALID_URL = r'https?://www\.groupon\.com/deals/(?P<id>[^?#]+)'
+ _VALID_URL = r'https?://(?:www\.)?groupon\.com/deals/(?P<id>[^/?#&]+)'
_TEST = {
'url': 'https://www.groupon.com/deals/bikram-yoga-huntington-beach-2#ooid=tubGNycTo_9Uxg82uESj4i61EYX8nyuf',
'description': 'Studio kept at 105 degrees and 40% humidity with anti-microbial and anti-slip Flotex flooring; certified instructors',
},
'playlist': [{
+ 'md5': '42428ce8a00585f9bc36e49226eae7a1',
'info_dict': {
- 'id': 'tubGNycTo_9Uxg82uESj4i61EYX8nyuf',
- 'ext': 'flv',
- 'title': 'Bikram Yoga Huntington Beach | Orange County',
+ 'id': 'fk6OhWpXgIQ',
+ 'ext': 'mp4',
+ 'title': 'Bikram Yoga Huntington Beach | Orange County !tubGNycTo@9Uxg82uESj4i61EYX8nyuf',
'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
- 'duration': 44.961,
+ 'duration': 45,
+ 'upload_date': '20160405',
+ 'uploader_id': 'groupon',
+ 'uploader': 'Groupon',
},
+ 'add_ie': ['Youtube'],
}],
'params': {
- 'skip_download': 'HDS',
- }
+ 'skip_download': True,
+ },
+ }
+
+ _PROVIDERS = {
+ 'ooyala': ('ooyala:%s', 'Ooyala'),
+ 'youtube': ('%s', 'Youtube'),
}
def _real_extract(self, url):
videos = payload['carousel'].get('dealVideos', [])
entries = []
for v in videos:
- if v.get('provider') != 'OOYALA':
+ provider = v.get('provider')
+ video_id = v.get('media') or v.get('id') or v.get('baseURL')
+ if not provider or not video_id:
+ continue
+ url_pattern, ie_key = self._PROVIDERS.get(provider.lower())
+ if not url_pattern:
self.report_warning(
'%s: Unsupported video provider %s, skipping video' %
- (playlist_id, v.get('provider')))
+ (playlist_id, provider))
continue
- entries.append(self.url_result('ooyala:%s' % v['media']))
+ entries.append(self.url_result(url_pattern % video_id, ie_key))
return {
'_type': 'playlist',
from ..compat import compat_urlparse
from ..utils import (
HEADRequest,
+ KNOWN_EXTENSIONS,
sanitized_Request,
str_to_int,
urlencode_postdata,
class HearThisAtIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?hearthis\.at/(?P<artist>[^/]+)/(?P<title>[A-Za-z0-9\-]+)/?$'
_PLAYLIST_URL = 'https://hearthis.at/playlist.php'
- _TEST = {
+ _TESTS = [{
'url': 'https://hearthis.at/moofi/dr-kreep',
'md5': 'ab6ec33c8fed6556029337c7885eb4e0',
'info_dict': {
'title': 'Moofi - Dr. Kreep',
'thumbnail': 're:^https?://.*\.jpg$',
'timestamp': 1421564134,
- 'description': 'Creepy Patch. Mutable Instruments Braids Vowel + Formant Mode.',
+ 'description': 'Listen to Dr. Kreep by Moofi on hearthis.at - Modular, Eurorack, Mutable Intruments Braids, Valhalla-DSP',
'upload_date': '20150118',
'comment_count': int,
'view_count': int,
'duration': 71,
'categories': ['Experimental'],
}
- }
+ }, {
+ # 'download' link redirects to the original webpage
+ 'url': 'https://hearthis.at/twitchsf/dj-jim-hopkins-totally-bitchin-80s-dance-mix/',
+ 'md5': '5980ceb7c461605d30f1f039df160c6e',
+ 'info_dict': {
+ 'id': '811296',
+ 'ext': 'mp3',
+ 'title': 'TwitchSF - DJ Jim Hopkins - Totally Bitchin\' 80\'s Dance Mix!',
+ 'description': 'Listen to DJ Jim Hopkins - Totally Bitchin\' 80\'s Dance Mix! by TwitchSF on hearthis.at - Dance',
+ 'upload_date': '20160328',
+ 'timestamp': 1459186146,
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ 'comment_count': int,
+ 'view_count': int,
+ 'like_count': int,
+ 'duration': 4360,
+ 'categories': ['Dance'],
+ },
+ }]
def _real_extract(self, url):
m = re.match(self._VALID_URL, url)
ext_handle = self._request_webpage(
ext_req, display_id, note='Determining extension')
ext = urlhandle_detect_ext(ext_handle)
- formats.append({
- 'format_id': 'download',
- 'vcodec': 'none',
- 'ext': ext,
- 'url': download_url,
- 'preference': 2, # Usually better quality
- })
+ if ext in KNOWN_EXTENSIONS:
+ formats.append({
+ 'format_id': 'download',
+ 'vcodec': 'none',
+ 'ext': ext,
+ 'url': download_url,
+ 'preference': 2, # Usually better quality
+ })
self._sort_formats(formats)
return {
_VALID_URL = r'https?://(?:www\.)?howcast\.com/videos/(?P<id>\d+)'
_TEST = {
'url': 'http://www.howcast.com/videos/390161-How-to-Tie-a-Square-Knot-Properly',
- 'md5': '8b743df908c42f60cf6496586c7f12c3',
+ 'md5': '7d45932269a288149483144f01b99789',
'info_dict': {
'id': '390161',
'ext': 'mp4',
'duration': 56.823,
},
'params': {
- # m3u8 download
'skip_download': True,
},
+ 'add_ie': ['Ooyala'],
}
def _real_extract(self, url):
from __future__ import unicode_literals
import re
-import json
from .common import InfoExtractor
from ..utils import (
+ mimetype2ext,
qualities,
)
class ImdbIE(InfoExtractor):
IE_NAME = 'imdb'
IE_DESC = 'Internet Movie Database trailers'
- _VALID_URL = r'https?://(?:www|m)\.imdb\.com/video/imdb/vi(?P<id>\d+)'
+ _VALID_URL = r'https?://(?:www|m)\.imdb\.com/video/[^/]+/vi(?P<id>\d+)'
- _TEST = {
+ _TESTS = [{
'url': 'http://www.imdb.com/video/imdb/vi2524815897',
'info_dict': {
'id': '2524815897',
'title': 'Ice Age: Continental Drift Trailer (No. 2) - IMDb',
'description': 'md5:9061c2219254e5d14e03c25c98e96a81',
}
- }
+ }, {
+ 'url': 'http://www.imdb.com/video/_/vi2524815897',
+ 'only_matching': True,
+ }]
def _real_extract(self, url):
video_id = self._match_id(url)
json_data = self._search_regex(
r'<script[^>]+class="imdb-player-data"[^>]*?>(.*?)</script>',
format_page, 'json data', flags=re.DOTALL)
- info = json.loads(json_data)
- format_info = info['videoPlayerObject']['video']
- f_id = format_info['ffname']
+ info = self._parse_json(json_data, video_id, fatal=False)
+ if not info:
+ continue
+ format_info = info.get('videoPlayerObject', {}).get('video', {})
+ if not format_info:
+ continue
+ video_info_list = format_info.get('videoInfoList')
+ if not video_info_list or not isinstance(video_info_list, list):
+ continue
+ video_info = video_info_list[0]
+ if not video_info or not isinstance(video_info, dict):
+ continue
+ video_url = video_info.get('videoUrl')
+ if not video_url:
+ continue
+ format_id = format_info.get('ffname')
formats.append({
- 'format_id': f_id,
- 'url': format_info['videoInfoList'][0]['videoUrl'],
- 'quality': quality(f_id),
+ 'format_id': format_id,
+ 'url': video_url,
+ 'ext': mimetype2ext(video_info.get('videoMimeType')),
+ 'quality': quality(format_id),
})
self._sort_formats(formats)
'enc': md5_text(enc_key + tail),
'qyid': _uuid,
'tn': random.random(),
- 'um': 0,
+ # In iQiyi's flash player, um is set to 1 if there's a logged user
+ # Some 1080P formats are only available with a logged user.
+ # Here force um=1 to trick the iQiyi server
+ 'um': 1,
'authkey': md5_text(md5_text('') + tail),
'k_tag': 1,
}
from .common import InfoExtractor
from ..utils import (
+ determine_ext,
float_or_none,
int_or_none,
)
class JWPlatformBaseIE(InfoExtractor):
- def _parse_jwplayer_data(self, jwplayer_data, video_id, require_title=True):
+ def _parse_jwplayer_data(self, jwplayer_data, video_id, require_title=True, m3u8_id=None, rtmp_params=None):
video_data = jwplayer_data['playlist'][0]
formats = []
for source in video_data['sources']:
source_url = self._proto_relative_url(source['file'])
source_type = source.get('type') or ''
- if source_type in ('application/vnd.apple.mpegurl', 'hls'):
+ if source_type in ('application/vnd.apple.mpegurl', 'hls') or determine_ext(source_url) == 'm3u8':
formats.extend(self._extract_m3u8_formats(
- source_url, video_id, 'mp4', 'm3u8_native', fatal=False))
+ source_url, video_id, 'mp4', 'm3u8_native', m3u8_id=m3u8_id, fatal=False))
elif source_type.startswith('audio'):
formats.append({
'url': source_url,
'vcodec': 'none',
})
else:
- formats.append({
+ a_format = {
'url': source_url,
'width': int_or_none(source.get('width')),
'height': int_or_none(source.get('height')),
- })
+ }
+ if source_url.startswith('rtmp'):
+ a_format['ext'] = 'flv',
+
+ # See com/longtailvideo/jwplayer/media/RTMPMediaProvider.as
+ # of jwplayer.flash.swf
+ rtmp_url_parts = re.split(
+ r'((?:mp4|mp3|flv):)', source_url, 1)
+ if len(rtmp_url_parts) == 3:
+ rtmp_url, prefix, play_path = rtmp_url_parts
+ a_format.update({
+ 'url': rtmp_url,
+ 'play_path': prefix + play_path,
+ })
+ if rtmp_params:
+ a_format.update(rtmp_params)
+ formats.append(a_format)
self._sort_formats(formats)
subtitles = {}
'id': '6446136',
'ext': 'mp3',
'title': '心',
- 'description': 'md5:b2ab6295d014005bfc607525bfc1e38a',
+ 'description': 'md5:5d0e947b242c35dc0eb1d2fce9fbf02c',
'creator': 'IU',
'upload_date': '20150518',
},
raise ExtractorError('this song has been offline because of copyright issues', expected=True)
song_name = self._html_search_regex(
- r'(?s)class="(?:[^"\s]+\s+)*title(?:\s+[^"\s]+)*".*?<h1[^>]+title="([^"]+)"', webpage, 'song name')
- singer_name = self._html_search_regex(
- r'<div[^>]+class="s_img">\s*<a[^>]+title="([^>]+)"',
- webpage, 'singer name', fatal=False)
+ r'<p[^>]+id="lrcName">([^<]+)</p>', webpage, 'song name')
+ singer_name = remove_start(self._html_search_regex(
+ r'<a[^>]+href="http://www\.kuwo\.cn/artist/content\?name=([^"]+)">',
+ webpage, 'singer name', fatal=False), '歌手')
lrc_content = clean_html(get_element_by_id('lrcContent', webpage))
if lrc_content == '暂无': # indicates no lyrics
lrc_content = None
self._sort_formats(formats)
album_id = self._html_search_regex(
- r'<p[^>]+class="album"[^<]+<a[^>]+href="http://www\.kuwo\.cn/album/(\d+)/"',
+ r'<a[^>]+href="http://www\.kuwo\.cn/album/(\d+)/"',
webpage, 'album id', fatal=False)
publish_time = None
'title': '八十年代精选',
'description': '这些都是属于八十年代的回忆！',
},
- 'playlist_count': 24,
+ 'playlist_mincount': 24,
}
def _real_extract(self, url):
category_desc = remove_start(
get_element_by_id('intro', webpage).strip(),
'%s简介：' % category_name)
+ if category_desc == '暂无':
+ category_desc = None
jsonm = self._parse_json(self._html_search_regex(
r'var\s+jsonm\s*=\s*([^;]+);', webpage, 'category songs'), category_id)
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class LearnrIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?learnr\.pro/view/video/(?P<id>[0-9]+)'
+ _TEST = {
+ 'url': 'http://www.learnr.pro/view/video/51624-web-development-tutorial-for-beginners-1-how-to-build-webpages-with-html-css-javascript',
+ 'md5': '3719fdf0a68397f49899e82c308a89de',
+ 'info_dict': {
+ 'id': '51624',
+ 'ext': 'mp4',
+ 'title': 'Web Development Tutorial for Beginners (#1) - How to build webpages with HTML, CSS, Javascript',
+ 'description': 'md5:b36dbfa92350176cdf12b4d388485503',
+ 'uploader': 'LearnCode.academy',
+ 'uploader_id': 'learncodeacademy',
+ 'upload_date': '20131021',
+ },
+ 'add_ie': ['Youtube'],
+ }
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ webpage = self._download_webpage(url, video_id)
+
+ return {
+ '_type': 'url_transparent',
+ 'url': self._search_regex(
+ r"videoId\s*:\s*'([^']+)'", webpage, 'youtube id'),
+ 'id': video_id,
+ }
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+
+from ..utils import (
+ determine_ext,
+ float_or_none,
+ int_or_none,
+ parse_filesize,
+)
+
+
+class LibraryOfCongressIE(InfoExtractor):
+ IE_NAME = 'loc'
+ IE_DESC = 'Library of Congress'
+ _VALID_URL = r'https?://(?:www\.)?loc\.gov/(?:item/|today/cyberlc/feature_wdesc\.php\?.*\brec=)(?P<id>[0-9]+)'
+ _TESTS = [{
+ # embedded via <div class="media-player"
+ 'url': 'http://loc.gov/item/90716351/',
+ 'md5': '353917ff7f0255aa6d4b80a034833de8',
+ 'info_dict': {
+ 'id': '90716351',
+ 'ext': 'mp4',
+ 'title': "Pa's trip to Mars",
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ 'duration': 0,
+ 'view_count': int,
+ },
+ }, {
+ # webcast embedded via mediaObjectId
+ 'url': 'https://www.loc.gov/today/cyberlc/feature_wdesc.php?rec=5578',
+ 'info_dict': {
+ 'id': '5578',
+ 'ext': 'mp4',
+ 'title': 'Help! Preservation Training Needs Here, There & Everywhere',
+ 'duration': 3765,
+ 'view_count': int,
+ 'subtitles': 'mincount:1',
+ },
+ 'params': {
+ 'skip_download': True,
+ },
+ }, {
+ # with direct download links
+ 'url': 'https://www.loc.gov/item/78710669/',
+ 'info_dict': {
+ 'id': '78710669',
+ 'ext': 'mp4',
+ 'title': 'La vie et la passion de Jesus-Christ',
+ 'duration': 0,
+ 'view_count': int,
+ 'formats': 'mincount:4',
+ },
+ 'params': {
+ 'skip_download': True,
+ },
+ }]
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ webpage = self._download_webpage(url, video_id)
+
+ media_id = self._search_regex(
+ (r'id=(["\'])media-player-(?P<id>.+?)\1',
+ r'<video[^>]+id=(["\'])uuid-(?P<id>.+?)\1',
+ r'<video[^>]+data-uuid=(["\'])(?P<id>.+?)\1',
+ r'mediaObjectId\s*:\s*(["\'])(?P<id>.+?)\1'),
+ webpage, 'media id', group='id')
+
+ data = self._download_json(
+ 'https://media.loc.gov/services/v1/media?id=%s&context=json' % media_id,
+ video_id)['mediaObject']
+
+ derivative = data['derivatives'][0]
+ media_url = derivative['derivativeUrl']
+
+ title = derivative.get('shortName') or data.get('shortName') or self._og_search_title(
+ webpage)
+
+ # Following algorithm was extracted from setAVSource js function
+ # found in webpage
+ media_url = media_url.replace('rtmp', 'https')
+
+ is_video = data.get('mediaType', 'v').lower() == 'v'
+ ext = determine_ext(media_url)
+ if ext not in ('mp4', 'mp3'):
+ media_url += '.mp4' if is_video else '.mp3'
+
+ if 'vod/mp4:' in media_url:
+ formats = [{
+ 'url': media_url.replace('vod/mp4:', 'hls-vod/media/') + '.m3u8',
+ 'format_id': 'hls',
+ 'ext': 'mp4',
+ 'protocol': 'm3u8_native',
+ 'quality': 1,
+ }]
+ elif 'vod/mp3:' in media_url:
+ formats = [{
+ 'url': media_url.replace('vod/mp3:', ''),
+ 'vcodec': 'none',
+ }]
+
+ download_urls = set()
+ for m in re.finditer(
+ r'<option[^>]+value=(["\'])(?P<url>.+?)\1[^>]+data-file-download=[^>]+>\s*(?P<id>.+?)(?:(?: |\s+)\((?P<size>.+?)\))?\s*<', webpage):
+ format_id = m.group('id').lower()
+ if format_id == 'gif':
+ continue
+ download_url = m.group('url')
+ if download_url in download_urls:
+ continue
+ download_urls.add(download_url)
+ formats.append({
+ 'url': download_url,
+ 'format_id': format_id,
+ 'filesize_approx': parse_filesize(m.group('size')),
+ })
+
+ self._sort_formats(formats)
+
+ duration = float_or_none(data.get('duration'))
+ view_count = int_or_none(data.get('viewCount'))
+
+ subtitles = {}
+ cc_url = data.get('ccUrl')
+ if cc_url:
+ subtitles.setdefault('en', []).append({
+ 'url': cc_url,
+ 'ext': 'ttml',
+ })
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'thumbnail': self._og_search_thumbnail(webpage, default=None),
+ 'duration': duration,
+ 'view_count': view_count,
+ 'formats': formats,
+ 'subtitles': subtitles,
+ }
from ..compat import compat_urlparse
from ..utils import (
determine_ext,
+ ExtractorError,
int_or_none,
+ parse_iso8601,
remove_end,
- unified_strdate,
- ExtractorError,
)
class LifeNewsIE(InfoExtractor):
- IE_NAME = 'lifenews'
- IE_DESC = 'LIFE | NEWS'
- _VALID_URL = r'https?://lifenews\.ru/(?:mobile/)?(?P<section>news|video)/(?P<id>\d+)'
+ IE_NAME = 'life'
+ IE_DESC = 'Life.ru'
+ _VALID_URL = r'https?://life\.ru/t/[^/]+/(?P<id>\d+)'
_TESTS = [{
# single video embedded via video/source
- 'url': 'http://lifenews.ru/news/98736',
+ 'url': 'https://life.ru/t/новости/98736',
'md5': '77c95eaefaca216e32a76a343ad89d23',
'info_dict': {
'id': '98736',
'ext': 'mp4',
'title': 'Мужчина нашел дома архив оборонного завода',
'description': 'md5:3b06b1b39b5e2bea548e403d99b8bf26',
+ 'timestamp': 1344154740,
'upload_date': '20120805',
+ 'view_count': int,
}
}, {
# single video embedded via iframe
- 'url': 'http://lifenews.ru/news/152125',
+ 'url': 'https://life.ru/t/новости/152125',
'md5': '77d19a6f0886cd76bdbf44b4d971a273',
'info_dict': {
'id': '152125',
'ext': 'mp4',
'title': 'В Сети появилось видео захвата «Правым сектором» колхозных полей ',
'description': 'Жители двух поселков Днепропетровской области не простили радикалам угрозу лишения плодородных земель и пошли в лобовую. ',
+ 'timestamp': 1427961840,
'upload_date': '20150402',
+ 'view_count': int,
}
}, {
# two videos embedded via iframe
- 'url': 'http://lifenews.ru/news/153461',
+ 'url': 'https://life.ru/t/новости/153461',
'info_dict': {
'id': '153461',
'title': 'В Москве спасли потерявшегося медвежонка, который спрятался на дереве',
'description': 'Маленький хищник не смог найти дорогу домой и обрел временное убежище на тополе недалеко от жилого массива, пока его не нашла соседская собака.',
- 'upload_date': '20150505',
+ 'timestamp': 1430825520,
+ 'view_count': int,
},
'playlist': [{
'md5': '9b6ef8bc0ffa25aebc8bdb40d89ab795',
'ext': 'mp4',
'title': 'В Москве спасли потерявшегося медвежонка, который спрятался на дереве (Видео 1)',
'description': 'Маленький хищник не смог найти дорогу домой и обрел временное убежище на тополе недалеко от жилого массива, пока его не нашла соседская собака.',
+ 'timestamp': 1430825520,
'upload_date': '20150505',
},
}, {
'ext': 'mp4',
'title': 'В Москве спасли потерявшегося медвежонка, который спрятался на дереве (Видео 2)',
'description': 'Маленький хищник не смог найти дорогу домой и обрел временное убежище на тополе недалеко от жилого массива, пока его не нашла соседская собака.',
+ 'timestamp': 1430825520,
'upload_date': '20150505',
},
}],
}, {
- 'url': 'http://lifenews.ru/video/13035',
+ 'url': 'https://life.ru/t/новости/213035',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://life.ru/t/%D0%BD%D0%BE%D0%B2%D0%BE%D1%81%D1%82%D0%B8/153461',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://life.ru/t/новости/411489/manuel_vals_nazval_frantsiiu_tsieliu_nomier_odin_dlia_ighil',
'only_matching': True,
}]
def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- video_id = mobj.group('id')
- section = mobj.group('section')
+ video_id = self._match_id(url)
- webpage = self._download_webpage(
- 'http://lifenews.ru/%s/%s' % (section, video_id),
- video_id, 'Downloading page')
+ webpage = self._download_webpage(url, video_id)
video_urls = re.findall(
r'<video[^>]+><source[^>]+src=["\'](.+?)["\']', webpage)
title = remove_end(
self._og_search_title(webpage),
- ' - Первый по срочным новостям — LIFE | NEWS')
+ ' - Life.ru')
description = self._og_search_description(webpage)
view_count = self._html_search_regex(
- r'<div class=\'views\'>\s*(\d+)\s*</div>', webpage, 'view count', fatal=False)
- comment_count = self._html_search_regex(
- r'=\'commentCount\'[^>]*>\s*(\d+)\s*<',
- webpage, 'comment count', fatal=False)
+ r'<div[^>]+class=(["\']).*?\bhits-count\b.*?\1[^>]*>\s*(?P<value>\d+)\s*</div>',
+ webpage, 'view count', fatal=False, group='value')
- upload_date = self._html_search_regex(
- r'<time[^>]*datetime=\'([^\']+)\'', webpage, 'upload date', fatal=False)
- if upload_date is not None:
- upload_date = unified_strdate(upload_date)
+ timestamp = parse_iso8601(self._search_regex(
+ r'<time[^>]+datetime=(["\'])(?P<value>.+?)\1',
+ webpage, 'upload date', fatal=False, group='value'))
common_info = {
'description': description,
'view_count': int_or_none(view_count),
- 'comment_count': int_or_none(comment_count),
- 'upload_date': upload_date,
+ 'timestamp': timestamp,
}
def make_entry(video_id, video_url, index=None):
ext = determine_ext(video_url)
if ext == 'm3u8':
formats.extend(self._extract_m3u8_formats(
- video_url, video_id, 'mp4', m3u8_id='m3u8'))
+ video_url, video_id, 'mp4',
+ entry_protocol='m3u8_native', m3u8_id='m3u8'))
else:
formats.append({
'url': video_url,
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import json
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+ ExtractorError,
+ int_or_none,
+ smuggle_url,
+ unsmuggle_url,
+)
+
+
+class LiTVIE(InfoExtractor):
+ _VALID_URL = r'https?://www\.litv\.tv/vod/[^/]+/content\.do\?.*?\bid=(?P<id>[^&]+)'
+
+ _URL_TEMPLATE = 'https://www.litv.tv/vod/%s/content.do?id=%s'
+
+ _TESTS = [{
+ 'url': 'https://www.litv.tv/vod/drama/content.do?brc_id=root&id=VOD00041610&isUHEnabled=true&autoPlay=1',
+ 'info_dict': {
+ 'id': 'VOD00041606',
+ 'title': '花千骨',
+ },
+ 'playlist_count': 50,
+ }, {
+ 'url': 'https://www.litv.tv/vod/drama/content.do?brc_id=root&id=VOD00041610&isUHEnabled=true&autoPlay=1',
+ 'info_dict': {
+ 'id': 'VOD00041610',
+ 'ext': 'mp4',
+ 'title': '花千骨第1集',
+ 'thumbnail': 're:https?://.*\.jpg$',
+ 'description': 'md5:c7017aa144c87467c4fb2909c4b05d6f',
+ 'episode_number': 1,
+ },
+ 'params': {
+ 'noplaylist': True,
+ 'skip_download': True, # m3u8 download
+ },
+ 'skip': 'Georestricted to Taiwan',
+ }]
+
+ def _extract_playlist(self, season_list, video_id, vod_data, view_data, prompt=True):
+ episode_title = view_data['title']
+ content_id = season_list['contentId']
+
+ if prompt:
+ self.to_screen('Downloading playlist %s - add --no-playlist to just download video %s' % (content_id, video_id))
+
+ all_episodes = [
+ self.url_result(smuggle_url(
+ self._URL_TEMPLATE % (view_data['contentType'], episode['contentId']),
+ {'force_noplaylist': True})) # To prevent infinite recursion
+ for episode in season_list['episode']]
+
+ return self.playlist_result(all_episodes, content_id, episode_title)
+
+ def _real_extract(self, url):
+ url, data = unsmuggle_url(url, {})
+
+ video_id = self._match_id(url)
+
+ noplaylist = self._downloader.params.get('noplaylist')
+ noplaylist_prompt = True
+ if 'force_noplaylist' in data:
+ noplaylist = data['force_noplaylist']
+ noplaylist_prompt = False
+
+ webpage = self._download_webpage(url, video_id)
+
+ view_data = dict(map(lambda t: (t[0], t[2]), re.findall(
+ r'viewData\.([a-zA-Z]+)\s*=\s*(["\'])([^"\']+)\2',
+ webpage)))
+
+ vod_data = self._parse_json(self._search_regex(
+ 'var\s+vod\s*=\s*([^;]+)', webpage, 'VOD data', default='{}'),
+ video_id)
+
+ season_list = list(vod_data.get('seasonList', {}).values())
+ if season_list:
+ if not noplaylist:
+ return self._extract_playlist(
+ season_list[0], video_id, vod_data, view_data,
+ prompt=noplaylist_prompt)
+
+ if noplaylist_prompt:
+ self.to_screen('Downloading just video %s because of --no-playlist' % video_id)
+
+ # In browsers `getMainUrl` request is always issued. Usually this
+ # endpoint gives the same result as the data embedded in the webpage.
+ # If georestricted, there are no embedded data, so an extra request is
+ # necessary to get the error code
+ video_data = self._parse_json(self._search_regex(
+ r'uiHlsUrl\s*=\s*testBackendData\(([^;]+)\);',
+ webpage, 'video data', default='{}'), video_id)
+ if not video_data:
+ payload = {
+ 'assetId': view_data['assetId'],
+ 'watchDevices': vod_data['watchDevices'],
+ 'contentType': view_data['contentType'],
+ }
+ video_data = self._download_json(
+ 'https://www.litv.tv/vod/getMainUrl', video_id,
+ data=json.dumps(payload).encode('utf-8'),
+ headers={'Content-Type': 'application/json'})
+
+ if not video_data.get('fullpath'):
+ error_msg = video_data.get('errorMessage')
+ if error_msg == 'vod.error.outsideregionerror':
+ self.raise_geo_restricted('This video is available in Taiwan only')
+ if error_msg:
+ raise ExtractorError('%s said: %s' % (self.IE_NAME, error_msg), expected=True)
+ raise ExtractorError('Unexpected result from %s' % self.IE_NAME)
+
+ formats = self._extract_m3u8_formats(
+ video_data['fullpath'], video_id, ext='mp4', m3u8_id='hls')
+ for a_format in formats:
+ # LiTV HLS segments doesn't like compressions
+ a_format.setdefault('http_headers', {})['Youtubedl-no-compression'] = True
+
+ title = view_data['title'] + view_data.get('secondaryMark', '')
+ description = view_data.get('description')
+ thumbnail = view_data.get('imageFile')
+ categories = [item['name'] for item in vod_data.get('category', [])]
+ episode = int_or_none(view_data.get('episode'))
+
+ return {
+ 'id': video_id,
+ 'formats': formats,
+ 'title': title,
+ 'description': description,
+ 'thumbnail': thumbnail,
+ 'categories': categories,
+ 'episode_number': episode,
+ }
'ext': 'flv',
'description': 'extremely bad day for this guy..!',
'uploader': 'ljfriel2',
- 'title': 'Most unlucky car accident'
+ 'title': 'Most unlucky car accident',
+ 'thumbnail': 're:^https?://.*\.jpg$'
}
}, {
'url': 'http://www.liveleak.com/view?i=f93_1390833151',
'description': 'German Television Channel NDR does an exclusive interview with Edward Snowden.\r\nUploaded on LiveLeak cause German Television thinks the rest of the world isn\'t intereseted in Edward Snowden.',
'uploader': 'ARD_Stinkt',
'title': 'German Television does first Edward Snowden Interview (ENGLISH)',
+ 'thumbnail': 're:^https?://.*\.jpg$'
}
}, {
'url': 'http://www.liveleak.com/view?i=4f7_1392687779',
'ext': 'mp4',
'description': 'Happened on 27.7.2014. \r\nAt 0:53 you can see people still swimming at near beach.',
'uploader': 'bony333',
- 'title': 'Crazy Hungarian tourist films close call waterspout in Croatia'
+ 'title': 'Crazy Hungarian tourist films close call waterspout in Croatia',
+ 'thumbnail': 're:^https?://.*\.jpg$'
}
}]
age_limit = int_or_none(self._search_regex(
r'you confirm that you are ([0-9]+) years and over.',
webpage, 'age limit', default=None))
+ video_thumbnail = self._og_search_thumbnail(webpage)
sources_raw = self._search_regex(
r'(?s)sources:\s*(\[.*?\]),', webpage, 'video URLs', default=None)
'uploader': video_uploader,
'formats': formats,
'age_limit': age_limit,
+ 'thumbnail': video_thumbnail,
}
}
def _extract_stream_info(self, stream_info):
- broadcast_id = stream_info['broadcast_id']
+ broadcast_id = compat_str(stream_info['broadcast_id'])
is_live = stream_info.get('is_live')
formats = []
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+
+
+class LocalNews8IE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?localnews8\.com/(?:[^/]+/)*(?P<display_id>[^/]+)/(?P<id>[0-9]+)'
+ _TEST = {
+ 'url': 'http://www.localnews8.com/news/rexburg-business-turns-carbon-fiber-scraps-into-wedding-rings/35183304',
+ 'md5': 'be4d48aea61aa2bde7be2ee47691ad20',
+ 'info_dict': {
+ 'id': '35183304',
+ 'display_id': 'rexburg-business-turns-carbon-fiber-scraps-into-wedding-rings',
+ 'ext': 'mp4',
+ 'title': 'Rexburg business turns carbon fiber scraps into wedding ring',
+ 'description': 'The process was first invented by Lamborghini and less than a dozen companies around the world use it.',
+ 'duration': 153,
+ 'timestamp': 1441844822,
+ 'upload_date': '20150910',
+ 'uploader_id': 'api',
+ }
+ }
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ video_id = mobj.group('id')
+ display_id = mobj.group('display_id')
+
+ webpage = self._download_webpage(url, display_id)
+
+ partner_id = self._search_regex(
+ r'partnerId\s*[:=]\s*(["\'])(?P<id>\d+)\1',
+ webpage, 'partner id', group='id')
+ kaltura_id = self._search_regex(
+ r'videoIdString\s*[:=]\s*(["\'])kaltura:(?P<id>[0-9a-z_]+)\1',
+ webpage, 'videl id', group='id')
+
+ return {
+ '_type': 'url_transparent',
+ 'url': 'kaltura:%s:%s' % (partner_id, kaltura_id),
+ 'ie_key': 'Kaltura',
+ 'id': video_id,
+ 'display_id': display_id,
+ }
+++ /dev/null
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..compat import compat_urllib_parse_unquote
-
-
-class MalemotionIE(InfoExtractor):
- _VALID_URL = r'https?://malemotion\.com/video/(.+?)\.(?P<id>.+?)(#|$)'
- _TEST = {
- 'url': 'http://malemotion.com/video/bete-de-concours.ltc',
- 'md5': '3013e53a0afbde2878bc39998c33e8a5',
- 'info_dict': {
- 'id': 'ltc',
- 'ext': 'mp4',
- 'title': 'Bête de Concours',
- 'age_limit': 18,
- },
- }
-
- def _real_extract(self, url):
- video_id = self._match_id(url)
- webpage = self._download_webpage(url, video_id)
-
- video_url = compat_urllib_parse_unquote(self._search_regex(
- r'<source type="video/mp4" src="(.+?)"', webpage, 'video URL'))
- video_title = self._html_search_regex(
- r'<title>(.*?)</title', webpage, 'title')
- video_thumbnail = self._search_regex(
- r'<video .+?poster="(.+?)"', webpage, 'thumbnail', fatal=False)
-
- formats = [{
- 'url': video_url,
- 'ext': 'mp4',
- 'format_id': 'mp4',
- 'preference': 1,
- }]
- self._sort_formats(formats)
-
- return {
- 'id': video_id,
- 'formats': formats,
- 'title': video_title,
- 'thumbnail': video_thumbnail,
- 'age_limit': 18,
- }
'title': 'Open: This is Face the Nation, February 9',
'description': 'md5:8a9ceec26d1f7ed6eab610834cc1a476',
'duration': 96,
+ 'uploader': 'CBSI-NEW',
+ 'upload_date': '20140209',
+ 'timestamp': 1391959800,
},
'params': {
# rtmp download
_TEST = {
'url': 'http://www.mgtv.com/v/1/290525/f/3116640.html',
- 'md5': '',
+ 'md5': '1bdadcf760a0b90946ca68ee9a2db41a',
'info_dict': {
'id': '3116640',
'ext': 'mp4',
'duration': 7461,
'thumbnail': 're:^https?://.*\.jpg$',
},
- 'params': {
- 'skip_download': True, # m3u8 download
- },
- }
-
- _FORMAT_MAP = {
- '标清': ('Standard', 0),
- '高清': ('High', 1),
- '超清': ('SuperHigh', 2),
}
def _real_extract(self, url):
formats = []
for idx, stream in enumerate(api_data['stream']):
- format_name = stream.get('name')
- format_id, preference = self._FORMAT_MAP.get(format_name, (None, None))
- format_info = self._download_json(
- stream['url'], video_id,
- note='Download video info for format %s' % format_id or '#%d' % idx)
- formats.append({
- 'format_id': format_id,
- 'url': format_info['info'],
- 'ext': 'mp4', # These are m3u8 playlists
- 'preference': preference,
- })
+ stream_url = stream.get('url')
+ if not stream_url:
+ continue
+ tbr = int_or_none(self._search_regex(
+ r'(\d+)\.mp4', stream_url, 'tbr', default=None))
+
+ def extract_format(stream_url, format_id, idx, query={}):
+ format_info = self._download_json(
+ stream_url, video_id,
+ note='Download video info for format %s' % format_id or '#%d' % idx, query=query)
+ return {
+ 'format_id': format_id,
+ 'url': format_info['info'],
+ 'ext': 'mp4',
+ 'tbr': tbr,
+ }
+
+ formats.append(extract_format(
+ stream_url, 'hls-%d' % tbr if tbr else None, idx * 2))
+ formats.append(extract_format(stream_url.replace(
+ '/playlist.m3u8', ''), 'http-%d' % tbr if tbr else None, idx * 2 + 1, {'pno': 1031}))
self._sort_formats(formats)
return {
--- /dev/null
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import (
+ compat_xpath,
+)
+from ..utils import (
+ int_or_none,
+ parse_duration,
+ smuggle_url,
+ unsmuggle_url,
+ xpath_text,
+)
+
+
+class MicrosoftVirtualAcademyBaseIE(InfoExtractor):
+ def _extract_base_url(self, course_id, display_id):
+ return self._download_json(
+ 'https://api-mlxprod.microsoft.com/services/products/anonymous/%s' % course_id,
+ display_id, 'Downloading course base URL')
+
+ def _extract_chapter_and_title(self, title):
+ if not title:
+ return None, None
+ m = re.search(r'(?P<chapter>\d+)\s*\|\s*(?P<title>.+)', title)
+ return (int(m.group('chapter')), m.group('title')) if m else (None, title)
+
+
+class MicrosoftVirtualAcademyIE(MicrosoftVirtualAcademyBaseIE):
+ IE_NAME = 'mva'
+ IE_DESC = 'Microsoft Virtual Academy videos'
+ _VALID_URL = r'(?:%s:|https?://(?:mva\.microsoft|(?:www\.)?microsoftvirtualacademy)\.com/[^/]+/training-courses/[^/?#&]+-)(?P<course_id>\d+)(?::|\?l=)(?P<id>[\da-zA-Z]+_\d+)' % IE_NAME
+
+ _TESTS = [{
+ 'url': 'https://mva.microsoft.com/en-US/training-courses/microsoft-azure-fundamentals-virtual-machines-11788?l=gfVXISmEB_6804984382',
+ 'md5': '7826c44fc31678b12ad8db11f6b5abb9',
+ 'info_dict': {
+ 'id': 'gfVXISmEB_6804984382',
+ 'ext': 'mp4',
+ 'title': 'Course Introduction',
+ 'formats': 'mincount:3',
+ 'subtitles': {
+ 'en': [{
+ 'ext': 'ttml',
+ }],
+ },
+ }
+ }, {
+ 'url': 'mva:11788:gfVXISmEB_6804984382',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ url, smuggled_data = unsmuggle_url(url, {})
+
+ mobj = re.match(self._VALID_URL, url)
+ course_id = mobj.group('course_id')
+ video_id = mobj.group('id')
+
+ base_url = smuggled_data.get('base_url') or self._extract_base_url(course_id, video_id)
+
+ settings = self._download_xml(
+ '%s/content/content_%s/videosettings.xml?v=1' % (base_url, video_id),
+ video_id, 'Downloading video settings XML')
+
+ _, title = self._extract_chapter_and_title(xpath_text(
+ settings, './/Title', 'title', fatal=True))
+
+ formats = []
+
+ for sources in settings.findall(compat_xpath('.//MediaSources')):
+ if sources.get('videoType') == 'smoothstreaming':
+ continue
+ for source in sources.findall(compat_xpath('./MediaSource')):
+ video_url = source.text
+ if not video_url or not video_url.startswith('http'):
+ continue
+ video_mode = source.get('videoMode')
+ height = int_or_none(self._search_regex(
+ r'^(\d+)[pP]$', video_mode or '', 'height', default=None))
+ codec = source.get('codec')
+ acodec, vcodec = [None] * 2
+ if codec:
+ codecs = codec.split(',')
+ if len(codecs) == 2:
+ acodec, vcodec = codecs
+ elif len(codecs) == 1:
+ vcodec = codecs[0]
+ formats.append({
+ 'url': video_url,
+ 'format_id': video_mode,
+ 'height': height,
+ 'acodec': acodec,
+ 'vcodec': vcodec,
+ })
+ self._sort_formats(formats)
+
+ subtitles = {}
+ for source in settings.findall(compat_xpath('.//MarkerResourceSource')):
+ subtitle_url = source.text
+ if not subtitle_url:
+ continue
+ subtitles.setdefault('en', []).append({
+ 'url': '%s/%s' % (base_url, subtitle_url),
+ 'ext': source.get('type'),
+ })
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'subtitles': subtitles,
+ 'formats': formats
+ }
+
+
+class MicrosoftVirtualAcademyCourseIE(MicrosoftVirtualAcademyBaseIE):
+ IE_NAME = 'mva:course'
+ IE_DESC = 'Microsoft Virtual Academy courses'
+ _VALID_URL = r'(?:%s:|https?://(?:mva\.microsoft|(?:www\.)?microsoftvirtualacademy)\.com/[^/]+/training-courses/(?P<display_id>[^/?#&]+)-)(?P<id>\d+)' % IE_NAME
+
+ _TESTS = [{
+ 'url': 'https://mva.microsoft.com/en-US/training-courses/microsoft-azure-fundamentals-virtual-machines-11788',
+ 'info_dict': {
+ 'id': '11788',
+ 'title': 'Microsoft Azure Fundamentals: Virtual Machines',
+ },
+ 'playlist_count': 36,
+ }, {
+ # with emphasized chapters
+ 'url': 'https://mva.microsoft.com/en-US/training-courses/developing-windows-10-games-with-construct-2-16335',
+ 'info_dict': {
+ 'id': '16335',
+ 'title': 'Developing Windows 10 Games with Construct 2',
+ },
+ 'playlist_count': 10,
+ }, {
+ 'url': 'https://www.microsoftvirtualacademy.com/en-US/training-courses/microsoft-azure-fundamentals-virtual-machines-11788',
+ 'only_matching': True,
+ }, {
+ 'url': 'mva:course:11788',
+ 'only_matching': True,
+ }]
+
+ @classmethod
+ def suitable(cls, url):
+ return False if MicrosoftVirtualAcademyIE.suitable(url) else super(
+ MicrosoftVirtualAcademyCourseIE, cls).suitable(url)
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ course_id = mobj.group('id')
+ display_id = mobj.group('display_id')
+
+ base_url = self._extract_base_url(course_id, display_id)
+
+ manifest = self._download_json(
+ '%s/imsmanifestlite.json' % base_url,
+ display_id, 'Downloading course manifest JSON')['manifest']
+
+ organization = manifest['organizations']['organization'][0]
+
+ entries = []
+ for chapter in organization['item']:
+ chapter_number, chapter_title = self._extract_chapter_and_title(chapter.get('title'))
+ chapter_id = chapter.get('@identifier')
+ for item in chapter.get('item', []):
+ item_id = item.get('@identifier')
+ if not item_id:
+ continue
+ metadata = item.get('resource', {}).get('metadata') or {}
+ if metadata.get('learningresourcetype') != 'Video':
+ continue
+ _, title = self._extract_chapter_and_title(item.get('title'))
+ duration = parse_duration(metadata.get('duration'))
+ description = metadata.get('description')
+ entries.append({
+ '_type': 'url_transparent',
+ 'url': smuggle_url(
+ 'mva:%s:%s' % (course_id, item_id), {'base_url': base_url}),
+ 'title': title,
+ 'description': description,
+ 'duration': duration,
+ 'chapter': chapter_title,
+ 'chapter_number': chapter_number,
+ 'chapter_id': chapter_id,
+ })
+
+ title = organization.get('title') or manifest.get('metadata', {}).get('title')
+
+ return self.playlist_result(entries, course_id, title)
IE_DESC = 'mitele.es'
_VALID_URL = r'https?://www\.mitele\.es/[^/]+/[^/]+/[^/]+/(?P<id>[^/]+)/'
- _TESTS = [{
+ _TEST = {
'url': 'http://www.mitele.es/programas-tv/diario-de/la-redaccion/programa-144/',
- 'md5': '0ff1a13aebb35d9bc14081ff633dd324',
+ # MD5 is unstable
'info_dict': {
'id': '0NF1jJnxS1Wu3pHrmvFyw2',
'display_id': 'programa-144',
'thumbnail': 're:(?i)^https?://.*\.jpg$',
'duration': 2913,
},
- }]
+ }
def _real_extract(self, url):
display_id = self._match_id(url)
+++ /dev/null
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..compat import compat_urllib_parse_urlencode
-
-
-class MuzuTVIE(InfoExtractor):
- _VALID_URL = r'https?://www\.muzu\.tv/(.+?)/(.+?)/(?P<id>\d+)'
- IE_NAME = 'muzu.tv'
-
- _TEST = {
- 'url': 'http://www.muzu.tv/defected/marcashken-featuring-sos-cat-walk-original-mix-music-video/1981454/',
- 'md5': '98f8b2c7bc50578d6a0364fff2bfb000',
- 'info_dict': {
- 'id': '1981454',
- 'ext': 'mp4',
- 'title': 'Cat Walk (Original Mix)',
- 'description': 'md5:90e868994de201b2570e4e5854e19420',
- 'uploader': 'MarcAshken featuring SOS',
- },
- }
-
- def _real_extract(self, url):
- video_id = self._match_id(url)
-
- info_data = compat_urllib_parse_urlencode({
- 'format': 'json',
- 'url': url,
- })
- info = self._download_json(
- 'http://www.muzu.tv/api/oembed/?%s' % info_data,
- video_id, 'Downloading video info')
-
- player_info = self._download_json(
- 'http://player.muzu.tv/player/playerInit?ai=%s' % video_id,
- video_id, 'Downloading player info')
- video_info = player_info['videos'][0]
- for quality in ['1080', '720', '480', '360']:
- if video_info.get('v%s' % quality):
- break
-
- data = compat_urllib_parse_urlencode({
- 'ai': video_id,
- # Even if each time you watch a video the hash changes,
- # it seems to work for different videos, and it will work
- # even if you use any non empty string as a hash
- 'viewhash': 'VBNff6djeV4HV5TRPW5kOHub2k',
- 'device': 'web',
- 'qv': quality,
- })
- video_url_info = self._download_json(
- 'http://player.muzu.tv/player/requestVideo?%s' % data,
- video_id, 'Downloading video url')
- video_url = video_url_info['url']
-
- return {
- 'id': video_id,
- 'title': info['title'],
- 'url': video_url,
- 'thumbnail': info['thumbnail_url'],
- 'description': info['description'],
- 'uploader': info['author_name'],
- }
class MwaveIE(InfoExtractor):
_VALID_URL = r'https?://mwave\.interest\.me/mnettv/videodetail\.m\?searchVideoDetailVO\.clip_id=(?P<id>[0-9]+)'
+ _URL_TEMPLATE = 'http://mwave.interest.me/mnettv/videodetail.m?searchVideoDetailVO.clip_id=%s'
_TEST = {
'url': 'http://mwave.interest.me/mnettv/videodetail.m?searchVideoDetailVO.clip_id=168859',
- 'md5': 'c930e27b7720aaa3c9d0018dfc8ff6cc',
+ # md5 is unstable
'info_dict': {
'id': '168859',
'ext': 'flv',
'view_count': int_or_none(vod_info.get('hit')),
'formats': formats,
}
+
+
+class MwaveMeetGreetIE(InfoExtractor):
+ _VALID_URL = r'https?://mwave\.interest\.me/meetgreet/view/(?P<id>\d+)'
+ _TEST = {
+ 'url': 'http://mwave.interest.me/meetgreet/view/256',
+ 'info_dict': {
+ 'id': '173294',
+ 'ext': 'flv',
+ 'title': '[MEET&GREET] Park BoRam',
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ 'uploader': 'Mwave',
+ 'duration': 3634,
+ 'view_count': int,
+ }
+ }
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ webpage = self._download_webpage(url, video_id)
+ clip_id = self._html_search_regex(
+ r'<iframe[^>]+src="/mnettv/ifr_clip\.m\?searchVideoDetailVO\.clip_id=(\d+)',
+ webpage, 'clip ID')
+ clip_url = MwaveIE._URL_TEMPLATE % clip_id
+ return self.url_result(clip_url, 'Mwave', clip_id)
'ext': 'flv',
'title': 'Tom Izzo, Michigan St. has \'so much respect\' for Duke',
'description': 'md5:ecb459c9d59e0766ac9c7d5d0eda8113',
+ 'uploader': 'NBCU-SPORTS',
+ 'upload_date': '20150330',
+ 'timestamp': 1427726529,
}
}
from __future__ import unicode_literals
-import re
-
from .common import InfoExtractor
from ..utils import (
- month_by_name,
int_or_none,
+ remove_end,
+ unified_strdate,
)
class NDTVIE(InfoExtractor):
- _VALID_URL = r'^https?://(?:www\.)?ndtv\.com/video/player/[^/]*/[^/]*/(?P<id>[a-z0-9]+)'
+ _VALID_URL = r'https?://(?:www\.)?ndtv\.com/video/(?:[^/]+/)+[^/?^&]+-(?P<id>\d+)'
_TEST = {
- 'url': 'http://www.ndtv.com/video/player/news/ndtv-exclusive-don-t-need-character-certificate-from-rahul-gandhi-says-arvind-kejriwal/300710',
+ 'url': 'http://www.ndtv.com/video/news/news/ndtv-exclusive-don-t-need-character-certificate-from-rahul-gandhi-says-arvind-kejriwal-300710',
'md5': '39f992dbe5fb531c395d8bbedb1e5e88',
'info_dict': {
'id': '300710',
'description': 'md5:ab2d4b4a6056c5cb4caa6d729deabf02',
'upload_date': '20131208',
'duration': 1327,
- 'thumbnail': 'http://i.ndtvimg.com/video/images/vod/medium/2013-12/big_300710_1386518307.jpg',
+ 'thumbnail': 're:https?://.*\.jpg',
},
}
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
+ title = remove_end(self._og_search_title(webpage), ' - NDTV')
+
filename = self._search_regex(
r"__filename='([^']+)'", webpage, 'video filename')
- video_url = ('http://bitcast-b.bitgravity.com/ndtvod/23372/ndtv/%s' %
- filename)
+ video_url = 'http://bitcast-b.bitgravity.com/ndtvod/23372/ndtv/%s' % filename
duration = int_or_none(self._search_regex(
r"__duration='([^']+)'", webpage, 'duration', fatal=False))
- date_m = re.search(r'''(?x)
- <p\s+class="vod_dateline">\s*
- Published\s+On:\s*
- (?P<monthname>[A-Za-z]+)\s+(?P<day>[0-9]+),\s*(?P<year>[0-9]+)
- ''', webpage)
- upload_date = None
-
- if date_m is not None:
- month = month_by_name(date_m.group('monthname'))
- if month is not None:
- upload_date = '%s%02d%02d' % (
- date_m.group('year'), month, int(date_m.group('day')))
-
- description = self._og_search_description(webpage)
- READ_MORE = ' (Read more)'
- if description.endswith(READ_MORE):
- description = description[:-len(READ_MORE)]
+ upload_date = unified_strdate(self._html_search_meta(
+ 'publish-date', webpage, 'upload date', fatal=False))
- title = self._og_search_title(webpage)
- TITLE_SUFFIX = ' - NDTV'
- if title.endswith(TITLE_SUFFIX):
- title = title[:-len(TITLE_SUFFIX)]
+ description = remove_end(self._og_search_description(webpage), ' (Read more)')
return {
'id': video_id,
from .common import InfoExtractor
from ..utils import (
- sanitized_Request,
+ clean_html,
+ determine_ext,
+ int_or_none,
+ qualities,
urlencode_postdata,
+ xpath_text,
)
'url': 'https://www.nfb.ca/film/qallunaat_why_white_people_are_funny',
'info_dict': {
'id': 'qallunaat_why_white_people_are_funny',
- 'ext': 'mp4',
+ 'ext': 'flv',
'title': 'Qallunaat! Why White People Are Funny ',
- 'description': 'md5:836d8aff55e087d04d9f6df554d4e038',
+ 'description': 'md5:6b8e32dde3abf91e58857b174916620c',
'duration': 3128,
+ 'creator': 'Mark Sandiford',
'uploader': 'Mark Sandiford',
- 'uploader_id': 'mark-sandiford',
},
'params': {
# rtmp download
def _real_extract(self, url):
video_id = self._match_id(url)
- page = self._download_webpage(
- 'https://www.nfb.ca/film/%s' % video_id, video_id,
- 'Downloading film page')
- uploader_id = self._html_search_regex(r'<a class="director-link" href="/explore-all-directors/([^/]+)/"',
- page, 'director id', fatal=False)
- uploader = self._html_search_regex(r'<em class="director-name" itemprop="name">([^<]+)</em>',
- page, 'director name', fatal=False)
-
- request = sanitized_Request(
+ config = self._download_xml(
'https://www.nfb.ca/film/%s/player_config' % video_id,
- urlencode_postdata({'getConfig': 'true'}))
- request.add_header('Content-Type', 'application/x-www-form-urlencoded')
- request.add_header('X-NFB-Referer', 'http://www.nfb.ca/medias/flash/NFBVideoPlayer.swf')
-
- config = self._download_xml(request, video_id, 'Downloading player config XML')
+ video_id, 'Downloading player config XML',
+ data=urlencode_postdata({'getConfig': 'true'}),
+ headers={
+ 'Content-Type': 'application/x-www-form-urlencoded',
+ 'X-NFB-Referer': 'http://www.nfb.ca/medias/flash/NFBVideoPlayer.swf'
+ })
- title = None
- description = None
- thumbnail = None
- duration = None
- formats = []
-
- def extract_thumbnail(media):
- thumbnails = {}
- for asset in media.findall('assets/asset'):
- thumbnails[asset.get('quality')] = asset.find('default/url').text
- if not thumbnails:
- return None
- if 'high' in thumbnails:
- return thumbnails['high']
- return list(thumbnails.values())[0]
+ title, description, thumbnail, duration, uploader, author = [None] * 6
+ thumbnails, formats = [[]] * 2
+ subtitles = {}
for media in config.findall('./player/stream/media'):
if media.get('type') == 'posterImage':
- thumbnail = extract_thumbnail(media)
+ quality_key = qualities(('low', 'high'))
+ thumbnails = []
+ for asset in media.findall('assets/asset'):
+ asset_url = xpath_text(asset, 'default/url', default=None)
+ if not asset_url:
+ continue
+ quality = asset.get('quality')
+ thumbnails.append({
+ 'url': asset_url,
+ 'id': quality,
+ 'preference': quality_key(quality),
+ })
elif media.get('type') == 'video':
- duration = int(media.get('duration'))
- title = media.find('title').text
- description = media.find('description').text
- # It seems assets always go from lower to better quality, so no need to sort
+ title = xpath_text(media, 'title', fatal=True)
for asset in media.findall('assets/asset'):
- for x in asset:
+ quality = asset.get('quality')
+ height = int_or_none(self._search_regex(
+ r'^(\d+)[pP]$', quality or '', 'height', default=None))
+ for node in asset:
+ streamer = xpath_text(node, 'streamerURI', default=None)
+ if not streamer:
+ continue
+ play_path = xpath_text(node, 'url', default=None)
+ if not play_path:
+ continue
formats.append({
- 'url': x.find('streamerURI').text,
- 'app': x.find('streamerURI').text.split('/', 3)[3],
- 'play_path': x.find('url').text,
+ 'url': streamer,
+ 'app': streamer.split('/', 3)[3],
+ 'play_path': play_path,
'rtmp_live': False,
- 'ext': 'mp4',
- 'format_id': '%s-%s' % (x.tag, asset.get('quality')),
+ 'ext': 'flv',
+ 'format_id': '%s-%s' % (node.tag, quality) if quality else node.tag,
+ 'height': height,
})
+ self._sort_formats(formats)
+ description = clean_html(xpath_text(media, 'description'))
+ uploader = xpath_text(media, 'author')
+ duration = int_or_none(media.get('duration'))
+ for subtitle in media.findall('./subtitles/subtitle'):
+ subtitle_url = xpath_text(subtitle, 'url', default=None)
+ if not subtitle_url:
+ continue
+ lang = xpath_text(subtitle, 'lang', default='en')
+ subtitles.setdefault(lang, []).append({
+ 'url': subtitle_url,
+ 'ext': (subtitle.get('format') or determine_ext(subtitle_url)).lower(),
+ })
return {
'id': video_id,
'title': title,
'description': description,
- 'thumbnail': thumbnail,
+ 'thumbnails': thumbnails,
'duration': duration,
+ 'creator': uploader,
'uploader': uploader,
- 'uploader_id': uploader_id,
'formats': formats,
+ 'subtitles': subtitles,
}
from __future__ import unicode_literals
from .common import InfoExtractor
+from .screenwavemedia import ScreenwaveMediaIE
from ..utils import (
unified_strdate,
_VALID_URL = r'https?://(?:www\.)?normalboots\.com/video/(?P<id>[0-9a-z-]*)/?$'
_TEST = {
'url': 'http://normalboots.com/video/home-alone-games-jontron/',
- 'md5': '8bf6de238915dd501105b44ef5f1e0f6',
'info_dict': {
'id': 'home-alone-games-jontron',
'ext': 'mp4',
'upload_date': '20140125',
},
'params': {
- # rtmp download
+ # m3u8 download
'skip_download': True,
},
+ 'add_ie': ['ScreenwaveMedia'],
}
def _real_extract(self, url):
r'<span style="text-transform:uppercase; font-size:inherit;">[A-Za-z]+, (?P<date>.*)</span>',
webpage, 'date', fatal=False))
- player_url = self._html_search_regex(
- r'<iframe\swidth="[0-9]+"\sheight="[0-9]+"\ssrc="(?P<url>[\S]+)"',
- webpage, 'player url')
- player_page = self._download_webpage(player_url, video_id)
- video_url = self._html_search_regex(
- r"file:\s'(?P<file>[^']+\.mp4)'", player_page, 'file')
+ screenwavemedia_url = self._html_search_regex(
+ ScreenwaveMediaIE.EMBED_PATTERN, webpage, 'screenwave URL',
+ group='url')
return {
+ '_type': 'url_transparent',
'id': video_id,
- 'url': video_url,
+ 'url': screenwavemedia_url,
+ 'ie_key': ScreenwaveMediaIE.ie_key(),
'title': self._og_search_title(webpage),
'description': self._og_search_description(webpage),
'thumbnail': self._og_search_thumbnail(webpage),
import re
from .common import InfoExtractor
-from ..compat import (
- compat_urlparse,
- compat_urllib_parse_unquote,
-)
+from ..compat import compat_urllib_parse_unquote
from ..utils import (
- determine_ext,
ExtractorError,
- float_or_none,
+ int_or_none,
+ parse_age_limit,
parse_duration,
- unified_strdate,
)
-class NRKIE(InfoExtractor):
- _VALID_URL = r'(?:nrk:|https?://(?:www\.)?nrk\.no/video/PS\*)(?P<id>\d+)'
-
- _TESTS = [
- {
- 'url': 'http://www.nrk.no/video/PS*150533',
- 'md5': 'bccd850baebefe23b56d708a113229c2',
- 'info_dict': {
- 'id': '150533',
- 'ext': 'flv',
- 'title': 'Dompap og andre fugler i Piip-Show',
- 'description': 'md5:d9261ba34c43b61c812cb6b0269a5c8f',
- 'duration': 263,
- }
- },
- {
- 'url': 'http://www.nrk.no/video/PS*154915',
- 'md5': '0b1493ba1aae7d9579a5ad5531bc395a',
- 'info_dict': {
- 'id': '154915',
- 'ext': 'flv',
- 'title': 'Slik høres internett ut når du er blind',
- 'description': 'md5:a621f5cc1bd75c8d5104cb048c6b8568',
- 'duration': 20,
- }
- },
- ]
+class NRKBaseIE(InfoExtractor):
+ def _extract_formats(self, manifest_url, video_id, fatal=True):
+ formats = []
+ formats.extend(self._extract_f4m_formats(
+ manifest_url + '?hdcore=3.5.0&plugin=aasp-3.5.0.151.81',
+ video_id, f4m_id='hds', fatal=fatal))
+ formats.extend(self._extract_m3u8_formats(manifest_url.replace(
+ 'akamaihd.net/z/', 'akamaihd.net/i/').replace('/manifest.f4m', '/master.m3u8'),
+ video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=fatal))
+ return formats
def _real_extract(self, url):
video_id = self._match_id(url)
data = self._download_json(
- 'http://v8.psapi.nrk.no/mediaelement/%s' % video_id,
- video_id, 'Downloading media JSON')
+ 'http://%s/mediaelement/%s' % (self._API_HOST, video_id),
+ video_id, 'Downloading mediaelement JSON')
+
+ title = data.get('fullTitle') or data.get('mainTitle') or data['title']
+ video_id = data.get('id') or video_id
+
+ entries = []
+
+ media_assets = data.get('mediaAssets')
+ if media_assets and isinstance(media_assets, list):
+ def video_id_and_title(idx):
+ return ((video_id, title) if len(media_assets) == 1
+ else ('%s-%d' % (video_id, idx), '%s (Part %d)' % (title, idx)))
+ for num, asset in enumerate(media_assets, 1):
+ asset_url = asset.get('url')
+ if not asset_url:
+ continue
+ formats = self._extract_formats(asset_url, video_id, fatal=False)
+ if not formats:
+ continue
+ self._sort_formats(formats)
+ entry_id, entry_title = video_id_and_title(num)
+ duration = parse_duration(asset.get('duration'))
+ subtitles = {}
+ for subtitle in ('webVtt', 'timedText'):
+ subtitle_url = asset.get('%sSubtitlesUrl' % subtitle)
+ if subtitle_url:
+ subtitles.setdefault('no', []).append({
+ 'url': compat_urllib_parse_unquote(subtitle_url)
+ })
+ entries.append({
+ 'id': asset.get('carrierId') or entry_id,
+ 'title': entry_title,
+ 'duration': duration,
+ 'subtitles': subtitles,
+ 'formats': formats,
+ })
- media_url = data.get('mediaUrl')
+ if not entries:
+ media_url = data.get('mediaUrl')
+ if media_url:
+ formats = self._extract_formats(media_url, video_id)
+ self._sort_formats(formats)
+ duration = parse_duration(data.get('duration'))
+ entries = [{
+ 'id': video_id,
+ 'title': title,
+ 'duration': duration,
+ 'formats': formats,
+ }]
- if not media_url:
- if data['usageRights']['isGeoBlocked']:
+ if not entries:
+ if data.get('usageRights', {}).get('isGeoBlocked'):
raise ExtractorError(
'NRK har ikke rettigheter til å vise dette programmet utenfor Norge',
expected=True)
- if determine_ext(media_url) == 'f4m':
- formats = self._extract_f4m_formats(
- media_url + '?hdcore=3.5.0&plugin=aasp-3.5.0.151.81', video_id, f4m_id='hds')
- self._sort_formats(formats)
- else:
- formats = [{
- 'url': media_url,
- 'ext': 'flv',
- }]
-
- duration = parse_duration(data.get('duration'))
+ conviva = data.get('convivaStatistics') or {}
+ series = conviva.get('seriesName') or data.get('seriesTitle')
+ episode = conviva.get('episodeName') or data.get('episodeNumberOrDate')
+ thumbnails = None
images = data.get('images')
- if images:
- thumbnails = images['webImages']
- thumbnails.sort(key=lambda image: image['pixelWidth'])
- thumbnail = thumbnails[-1]['imageUrl']
- else:
- thumbnail = None
-
- return {
- 'id': video_id,
- 'title': data['title'],
- 'description': data['description'],
- 'duration': duration,
- 'thumbnail': thumbnail,
- 'formats': formats,
+ if images and isinstance(images, dict):
+ web_images = images.get('webImages')
+ if isinstance(web_images, list):
+ thumbnails = [{
+ 'url': image['imageUrl'],
+ 'width': int_or_none(image.get('width')),
+ 'height': int_or_none(image.get('height')),
+ } for image in web_images if image.get('imageUrl')]
+
+ description = data.get('description')
+
+ common_info = {
+ 'description': description,
+ 'series': series,
+ 'episode': episode,
+ 'age_limit': parse_age_limit(data.get('legalAge')),
+ 'thumbnails': thumbnails,
}
+ vcodec = 'none' if data.get('mediaType') == 'Audio' else None
+
+ # TODO: extract chapters when https://github.com/rg3/youtube-dl/pull/9409 is merged
+
+ for entry in entries:
+ entry.update(common_info)
+ for f in entry['formats']:
+ f['vcodec'] = vcodec
+
+ return self.playlist_result(entries, video_id, title, description)
+
+
+class NRKIE(NRKBaseIE):
+ _VALID_URL = r'(?:nrk:|https?://(?:www\.)?nrk\.no/video/PS\*)(?P<id>\d+)'
+ _API_HOST = 'v8.psapi.nrk.no'
+ _TESTS = [{
+ # video
+ 'url': 'http://www.nrk.no/video/PS*150533',
+ 'md5': '2f7f6eeb2aacdd99885f355428715cfa',
+ 'info_dict': {
+ 'id': '150533',
+ 'ext': 'mp4',
+ 'title': 'Dompap og andre fugler i Piip-Show',
+ 'description': 'md5:d9261ba34c43b61c812cb6b0269a5c8f',
+ 'duration': 263,
+ }
+ }, {
+ # audio
+ 'url': 'http://www.nrk.no/video/PS*154915',
+ # MD5 is unstable
+ 'info_dict': {
+ 'id': '154915',
+ 'ext': 'flv',
+ 'title': 'Slik høres internett ut når du er blind',
+ 'description': 'md5:a621f5cc1bd75c8d5104cb048c6b8568',
+ 'duration': 20,
+ }
+ }]
+
+
+class NRKTVIE(NRKBaseIE):
+ IE_DESC = 'NRK TV and NRK Radio'
+ _VALID_URL = r'https?://(?:tv|radio)\.nrk(?:super)?\.no/(?:serie/[^/]+|program)/(?P<id>[a-zA-Z]{4}\d{8})(?:/\d{2}-\d{2}-\d{4})?(?:#del=(?P<part_id>\d+))?'
+ _API_HOST = 'psapi-we.nrk.no'
+
+ _TESTS = [{
+ 'url': 'https://tv.nrk.no/serie/20-spoersmaal-tv/MUHH48000314/23-05-2014',
+ 'md5': '4e9ca6629f09e588ed240fb11619922a',
+ 'info_dict': {
+ 'id': 'MUHH48000314AA',
+ 'ext': 'mp4',
+ 'title': '20 spørsmål 23.05.2014',
+ 'description': 'md5:bdea103bc35494c143c6a9acdd84887a',
+ 'duration': 1741.52,
+ },
+ }, {
+ 'url': 'https://tv.nrk.no/program/mdfp15000514',
+ 'md5': '43d0be26663d380603a9cf0c24366531',
+ 'info_dict': {
+ 'id': 'MDFP15000514CA',
+ 'ext': 'mp4',
+ 'title': 'Grunnlovsjubiléet - Stor ståhei for ingenting 24.05.2014',
+ 'description': 'md5:89290c5ccde1b3a24bb8050ab67fe1db',
+ 'duration': 4605.08,
+ },
+ }, {
+ # single playlist video
+ 'url': 'https://tv.nrk.no/serie/tour-de-ski/MSPO40010515/06-01-2015#del=2',
+ 'md5': 'adbd1dbd813edaf532b0a253780719c2',
+ 'info_dict': {
+ 'id': 'MSPO40010515-part2',
+ 'ext': 'flv',
+ 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn 06.01.2015 (del 2:2)',
+ 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
+ },
+ 'skip': 'Only works from Norway',
+ }, {
+ 'url': 'https://tv.nrk.no/serie/tour-de-ski/MSPO40010515/06-01-2015',
+ 'playlist': [{
+ 'md5': '9480285eff92d64f06e02a5367970a7a',
+ 'info_dict': {
+ 'id': 'MSPO40010515-part1',
+ 'ext': 'flv',
+ 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn 06.01.2015 (del 1:2)',
+ 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
+ },
+ }, {
+ 'md5': 'adbd1dbd813edaf532b0a253780719c2',
+ 'info_dict': {
+ 'id': 'MSPO40010515-part2',
+ 'ext': 'flv',
+ 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn 06.01.2015 (del 2:2)',
+ 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
+ },
+ }],
+ 'info_dict': {
+ 'id': 'MSPO40010515',
+ 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn',
+ 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
+ 'duration': 6947.52,
+ },
+ 'skip': 'Only works from Norway',
+ }, {
+ 'url': 'https://radio.nrk.no/serie/dagsnytt/NPUB21019315/12-07-2015#',
+ 'only_matching': True,
+ }]
+
class NRKPlaylistIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?nrk\.no/(?!video|skole)(?:[^/]+/)+(?P<id>[^/]+)'
nrk_id = self._search_regex(r'data-nrk-id=["\'](\d+)', webpage, 'nrk id')
return self.url_result('nrk:%s' % nrk_id)
-
-
-class NRKTVIE(InfoExtractor):
- IE_DESC = 'NRK TV and NRK Radio'
- _VALID_URL = r'(?P<baseurl>https?://(?:tv|radio)\.nrk(?:super)?\.no/)(?:serie/[^/]+|program)/(?P<id>[a-zA-Z]{4}\d{8})(?:/\d{2}-\d{2}-\d{4})?(?:#del=(?P<part_id>\d+))?'
-
- _TESTS = [
- {
- 'url': 'https://tv.nrk.no/serie/20-spoersmaal-tv/MUHH48000314/23-05-2014',
- 'info_dict': {
- 'id': 'MUHH48000314',
- 'ext': 'mp4',
- 'title': '20 spørsmål',
- 'description': 'md5:bdea103bc35494c143c6a9acdd84887a',
- 'upload_date': '20140523',
- 'duration': 1741.52,
- },
- 'params': {
- # m3u8 download
- 'skip_download': True,
- },
- },
- {
- 'url': 'https://tv.nrk.no/program/mdfp15000514',
- 'info_dict': {
- 'id': 'mdfp15000514',
- 'ext': 'mp4',
- 'title': 'Grunnlovsjubiléet - Stor ståhei for ingenting',
- 'description': 'md5:654c12511f035aed1e42bdf5db3b206a',
- 'upload_date': '20140524',
- 'duration': 4605.08,
- },
- 'params': {
- # m3u8 download
- 'skip_download': True,
- },
- },
- {
- # single playlist video
- 'url': 'https://tv.nrk.no/serie/tour-de-ski/MSPO40010515/06-01-2015#del=2',
- 'md5': 'adbd1dbd813edaf532b0a253780719c2',
- 'info_dict': {
- 'id': 'MSPO40010515-part2',
- 'ext': 'flv',
- 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn 06.01.2015 (del 2:2)',
- 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
- 'upload_date': '20150106',
- },
- 'skip': 'Only works from Norway',
- },
- {
- 'url': 'https://tv.nrk.no/serie/tour-de-ski/MSPO40010515/06-01-2015',
- 'playlist': [
- {
- 'md5': '9480285eff92d64f06e02a5367970a7a',
- 'info_dict': {
- 'id': 'MSPO40010515-part1',
- 'ext': 'flv',
- 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn 06.01.2015 (del 1:2)',
- 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
- 'upload_date': '20150106',
- },
- },
- {
- 'md5': 'adbd1dbd813edaf532b0a253780719c2',
- 'info_dict': {
- 'id': 'MSPO40010515-part2',
- 'ext': 'flv',
- 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn 06.01.2015 (del 2:2)',
- 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
- 'upload_date': '20150106',
- },
- },
- ],
- 'info_dict': {
- 'id': 'MSPO40010515',
- 'title': 'Tour de Ski: Sprint fri teknikk, kvinner og menn',
- 'description': 'md5:238b67b97a4ac7d7b4bf0edf8cc57d26',
- 'upload_date': '20150106',
- 'duration': 6947.5199999999995,
- },
- 'skip': 'Only works from Norway',
- },
- {
- 'url': 'https://radio.nrk.no/serie/dagsnytt/NPUB21019315/12-07-2015#',
- 'only_matching': True,
- }
- ]
-
- def _extract_f4m(self, manifest_url, video_id):
- return self._extract_f4m_formats(
- manifest_url + '?hdcore=3.1.1&plugin=aasp-3.1.1.69.124', video_id, f4m_id='hds')
-
- def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- video_id = mobj.group('id')
- part_id = mobj.group('part_id')
- base_url = mobj.group('baseurl')
-
- webpage = self._download_webpage(url, video_id)
-
- title = self._html_search_meta(
- 'title', webpage, 'title')
- description = self._html_search_meta(
- 'description', webpage, 'description')
-
- thumbnail = self._html_search_regex(
- r'data-posterimage="([^"]+)"',
- webpage, 'thumbnail', fatal=False)
- upload_date = unified_strdate(self._html_search_meta(
- 'rightsfrom', webpage, 'upload date', fatal=False))
- duration = float_or_none(self._html_search_regex(
- r'data-duration="([^"]+)"',
- webpage, 'duration', fatal=False))
-
- # playlist
- parts = re.findall(
- r'<a href="#del=(\d+)"[^>]+data-argument="([^"]+)">([^<]+)</a>', webpage)
- if parts:
- entries = []
- for current_part_id, stream_url, part_title in parts:
- if part_id and current_part_id != part_id:
- continue
- video_part_id = '%s-part%s' % (video_id, current_part_id)
- formats = self._extract_f4m(stream_url, video_part_id)
- entries.append({
- 'id': video_part_id,
- 'title': part_title,
- 'description': description,
- 'thumbnail': thumbnail,
- 'upload_date': upload_date,
- 'formats': formats,
- })
- if part_id:
- if entries:
- return entries[0]
- else:
- playlist = self.playlist_result(entries, video_id, title, description)
- playlist.update({
- 'thumbnail': thumbnail,
- 'upload_date': upload_date,
- 'duration': duration,
- })
- return playlist
-
- formats = []
-
- f4m_url = re.search(r'data-media="([^"]+)"', webpage)
- if f4m_url:
- formats.extend(self._extract_f4m(f4m_url.group(1), video_id))
-
- m3u8_url = re.search(r'data-hls-media="([^"]+)"', webpage)
- if m3u8_url:
- formats.extend(self._extract_m3u8_formats(m3u8_url.group(1), video_id, 'mp4', m3u8_id='hls'))
- self._sort_formats(formats)
-
- subtitles_url = self._html_search_regex(
- r'data-subtitlesurl\s*=\s*(["\'])(?P<url>.+?)\1',
- webpage, 'subtitle URL', default=None, group='url')
- subtitles = {}
- if subtitles_url:
- subtitles['no'] = [{
- 'ext': 'ttml',
- 'url': compat_urlparse.urljoin(base_url, subtitles_url),
- }]
-
- return {
- 'id': video_id,
- 'title': title,
- 'description': description,
- 'thumbnail': thumbnail,
- 'upload_date': upload_date,
- 'duration': duration,
- 'formats': formats,
- 'subtitles': subtitles,
- }
from .common import InfoExtractor
from ..utils import (
parse_duration,
- sanitized_Request,
- unified_strdate,
)
'ext': 'mp4',
'title': 'Horny babes show their awesome bodeis and',
'duration': 129,
- 'upload_date': '20140508',
'age_limit': 18,
}
}
def _real_extract(self, url):
video_id = self._match_id(url)
- formats = []
+ page_url = 'http://m.nuvid.com/video/%s' % video_id
+ webpage = self._download_webpage(
+ page_url, video_id, 'Downloading video page')
+ # When dwnld_speed exists and has a value larger than the MP4 file's
+ # bitrate, Nuvid returns the MP4 URL
+ # It's unit is 100bytes/millisecond, see mobile-nuvid-min.js for the algorithm
+ self._set_cookie('nuvid.com', 'dwnld_speed', '10.0')
+ mp4_webpage = self._download_webpage(
+ page_url, video_id, 'Downloading video page for MP4 format')
- for dwnld_speed, format_id in [(0, '3gp'), (5, 'mp4')]:
- request = sanitized_Request(
- 'http://m.nuvid.com/play/%s' % video_id)
- request.add_header('Cookie', 'skip_download_page=1; dwnld_speed=%d; adv_show=1' % dwnld_speed)
- webpage = self._download_webpage(
- request, video_id, 'Downloading %s page' % format_id)
- video_url = self._html_search_regex(
- r'<a\s+href="([^"]+)"\s+class="b_link">', webpage, '%s video URL' % format_id, fatal=False)
- if not video_url:
- continue
+ html5_video_re = r'(?s)<(?:video|audio)[^<]*(?:>.*?<source[^>]*)?\s+src=["\'](.*?)["\']',
+ video_url = self._html_search_regex(html5_video_re, webpage, video_id)
+ mp4_video_url = self._html_search_regex(html5_video_re, mp4_webpage, video_id)
+ formats = [{
+ 'url': video_url,
+ }]
+ if mp4_video_url != video_url:
formats.append({
- 'url': video_url,
- 'format_id': format_id,
+ 'url': mp4_video_url,
})
- webpage = self._download_webpage(
- 'http://m.nuvid.com/video/%s' % video_id, video_id, 'Downloading video page')
title = self._html_search_regex(
[r'<span title="([^"]+)">',
- r'<div class="thumb-holder video">\s*<h5[^>]*>([^<]+)</h5>'], webpage, 'title').strip()
+ r'<div class="thumb-holder video">\s*<h5[^>]*>([^<]+)</h5>',
+ r'<span[^>]+class="title_thumb">([^<]+)</span>'], webpage, 'title').strip()
thumbnails = [
{
'url': thumb_url,
]
thumbnail = thumbnails[0]['url'] if thumbnails else None
duration = parse_duration(self._html_search_regex(
- r'<i class="fa fa-clock-o"></i>\s*(\d{2}:\d{2})', webpage, 'duration', fatal=False))
- upload_date = unified_strdate(self._html_search_regex(
- r'<i class="fa fa-user"></i>\s*(\d{4}-\d{2}-\d{2})', webpage, 'upload date', fatal=False))
+ [r'<i class="fa fa-clock-o"></i>\s*(\d{2}:\d{2})',
+ r'<span[^>]+class="view_time">([^<]+)</span>'], webpage, 'duration', fatal=False))
return {
'id': video_id,
'thumbnails': thumbnails,
'thumbnail': thumbnail,
'duration': duration,
- 'upload_date': upload_date,
'age_limit': 18,
'formats': formats,
}
from __future__ import unicode_literals
from .common import InfoExtractor
-from ..compat import compat_urllib_parse_unquote
+from ..compat import (
+ compat_parse_qs,
+ compat_urllib_parse_unquote,
+ compat_urllib_parse_urlparse,
+)
from ..utils import (
ExtractorError,
unified_strdate,
'skip': 'Video has been blocked',
}, {
# metadataUrl
- 'url': 'http://ok.ru/video/63567059965189-0',
+ 'url': 'http://ok.ru/video/63567059965189-0?fromTime=5',
'md5': '9676cf86eff5391d35dea675d224e131',
'info_dict': {
'id': '63567059965189-0',
'uploader': '☭ Андрей Мещанинов ☭',
'like_count': int,
'age_limit': 0,
+ 'start_time': 5,
},
}, {
# YouTube embed (metadataUrl, provider == USER_YOUTUBE)
'uploader': 'Алина П',
'age_limit': 0,
},
+ }, {
+ # YouTube embed (metadata, provider == USER_YOUTUBE, no metadata.movie.title field)
+ 'url': 'http://ok.ru/video/62036049272859-0',
+ 'info_dict': {
+ 'id': '62036049272859-0',
+ 'ext': 'mp4',
+ 'title': 'МУЗЫКА ДОЖДЯ .',
+ 'description': 'md5:6f1867132bd96e33bf53eda1091e8ed0',
+ 'upload_date': '20120106',
+ 'uploader_id': '473534735899',
+ 'uploader': 'МARINA D',
+ 'age_limit': 0,
+ },
+ 'params': {
+ 'skip_download': True,
+ },
}, {
'url': 'http://ok.ru/web-api/video/moviePlayer/20079905452',
'only_matching': True,
}]
def _real_extract(self, url):
+ start_time = int_or_none(compat_parse_qs(
+ compat_urllib_parse_urlparse(url).query).get('fromTime', [None])[0])
+
video_id = self._match_id(url)
webpage = self._download_webpage(
video_id, 'Downloading metadata JSON')
movie = metadata['movie']
- title = movie['title']
+
+ # Some embedded videos may not contain title in movie dict (e.g.
+ # http://ok.ru/video/62036049272859-0) thus we allow missing title
+ # here and it's going to be extracted later by an extractor that
+ # will process the actual embed.
+ provider = metadata.get('provider')
+ title = movie['title'] if provider == 'UPLOADED_ODKL' else movie.get('title')
+
thumbnail = movie.get('poster')
duration = int_or_none(movie.get('duration'))
'uploader_id': uploader_id,
'like_count': like_count,
'age_limit': age_limit,
+ 'start_time': start_time,
}
- if metadata.get('provider') == 'USER_YOUTUBE':
+ if provider == 'USER_YOUTUBE':
info.update({
'_type': 'url_transparent',
'url': movie['contentId'],
r'share_title\s*=\s*(["\'])(?P<title>[^\1]+?)\1',
webpage, 'title', group='title')
description = self._search_regex(
- r'share_description\s*=\s*(["\'])(?P<description>[^\1]+?)\1',
+ r'share_description\s*=\s*(["\'])(?P<description>[^\'"]+?)\1',
webpage, 'description', default=None, group='description')
thumbnail = self._search_regex(
r'poster\s*=\s*(["\'])(?P<thumbnail>[^\1]+?)\1',
float_or_none,
ExtractorError,
unsmuggle_url,
+ determine_ext,
)
from ..compat import compat_urllib_parse_urlencode
class OoyalaBaseIE(InfoExtractor):
_PLAYER_BASE = 'http://player.ooyala.com/'
_CONTENT_TREE_BASE = _PLAYER_BASE + 'player_api/v1/content_tree/'
- _AUTHORIZATION_URL_TEMPLATE = _PLAYER_BASE + 'sas/player_api/v1/authorization/embed_code/%s/%s?'
+ _AUTHORIZATION_URL_TEMPLATE = _PLAYER_BASE + 'sas/player_api/v2/authorization/embed_code/%s/%s?'
def _extract(self, content_tree_url, video_id, domain='example.org'):
content_tree = self._download_json(content_tree_url, video_id)['content_tree']
metadata = content_tree[list(content_tree)[0]]
embed_code = metadata['embed_code']
pcode = metadata.get('asset_pcode') or embed_code
- video_info = {
- 'id': embed_code,
- 'title': metadata['title'],
- 'description': metadata.get('description'),
- 'thumbnail': metadata.get('thumbnail_image') or metadata.get('promo_image'),
- 'duration': float_or_none(metadata.get('duration'), 1000),
- }
+ title = metadata['title']
+
+ auth_data = self._download_json(
+ self._AUTHORIZATION_URL_TEMPLATE % (pcode, embed_code) +
+ compat_urllib_parse_urlencode({
+ 'domain': domain,
+ 'supportedFormats': 'mp4,rtmp,m3u8,hds',
+ }), video_id)
+
+ cur_auth_data = auth_data['authorization_data'][embed_code]
urls = []
formats = []
- for supported_format in ('mp4', 'm3u8', 'hds', 'rtmp'):
- auth_data = self._download_json(
- self._AUTHORIZATION_URL_TEMPLATE % (pcode, embed_code) +
- compat_urllib_parse_urlencode({
- 'domain': domain,
- 'supportedFormats': supported_format
- }),
- video_id, 'Downloading %s JSON' % supported_format)
-
- cur_auth_data = auth_data['authorization_data'][embed_code]
-
- if cur_auth_data['authorized']:
- for stream in cur_auth_data['streams']:
- url = base64.b64decode(
- stream['url']['data'].encode('ascii')).decode('utf-8')
- if url in urls:
- continue
- urls.append(url)
- delivery_type = stream['delivery_type']
- if delivery_type == 'hls' or '.m3u8' in url:
- formats.extend(self._extract_m3u8_formats(
- url, embed_code, 'mp4', 'm3u8_native',
- m3u8_id='hls', fatal=False))
- elif delivery_type == 'hds' or '.f4m' in url:
- formats.extend(self._extract_f4m_formats(
- url + '?hdcore=3.7.0', embed_code, f4m_id='hds', fatal=False))
- elif '.smil' in url:
- formats.extend(self._extract_smil_formats(
- url, embed_code, fatal=False))
- else:
- formats.append({
- 'url': url,
- 'ext': stream.get('delivery_type'),
- 'vcodec': stream.get('video_codec'),
- 'format_id': delivery_type,
- 'width': int_or_none(stream.get('width')),
- 'height': int_or_none(stream.get('height')),
- 'abr': int_or_none(stream.get('audio_bitrate')),
- 'vbr': int_or_none(stream.get('video_bitrate')),
- 'fps': float_or_none(stream.get('framerate')),
- })
- else:
- raise ExtractorError('%s said: %s' % (
- self.IE_NAME, cur_auth_data['message']), expected=True)
+ if cur_auth_data['authorized']:
+ for stream in cur_auth_data['streams']:
+ s_url = base64.b64decode(
+ stream['url']['data'].encode('ascii')).decode('utf-8')
+ if s_url in urls:
+ continue
+ urls.append(s_url)
+ ext = determine_ext(s_url, None)
+ delivery_type = stream['delivery_type']
+ if delivery_type == 'hls' or ext == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ s_url, embed_code, 'mp4', 'm3u8_native',
+ m3u8_id='hls', fatal=False))
+ elif delivery_type == 'hds' or ext == 'f4m':
+ formats.extend(self._extract_f4m_formats(
+ s_url + '?hdcore=3.7.0', embed_code, f4m_id='hds', fatal=False))
+ elif ext == 'smil':
+ formats.extend(self._extract_smil_formats(
+ s_url, embed_code, fatal=False))
+ else:
+ formats.append({
+ 'url': s_url,
+ 'ext': ext or stream.get('delivery_type'),
+ 'vcodec': stream.get('video_codec'),
+ 'format_id': delivery_type,
+ 'width': int_or_none(stream.get('width')),
+ 'height': int_or_none(stream.get('height')),
+ 'abr': int_or_none(stream.get('audio_bitrate')),
+ 'vbr': int_or_none(stream.get('video_bitrate')),
+ 'fps': float_or_none(stream.get('framerate')),
+ })
+ else:
+ raise ExtractorError('%s said: %s' % (
+ self.IE_NAME, cur_auth_data['message']), expected=True)
self._sort_formats(formats)
- video_info['formats'] = formats
- return video_info
+ subtitles = {}
+ for lang, sub in metadata.get('closed_captions_vtt', {}).get('captions', {}).items():
+ sub_url = sub.get('url')
+ if not sub_url:
+ continue
+ subtitles[lang] = [{
+ 'url': sub_url,
+ }]
+
+ return {
+ 'id': embed_code,
+ 'title': title,
+ 'description': metadata.get('description'),
+ 'thumbnail': metadata.get('thumbnail_image') or metadata.get('promo_image'),
+ 'duration': float_or_none(metadata.get('duration'), 1000),
+ 'subtitles': subtitles,
+ 'formats': formats,
+ }
class OoyalaIE(OoyalaBaseIE):
'description': 'How badly damaged does a drive have to be to defeat Russell and his crew? Apparently, smashed to bits.',
'duration': 853.386,
},
+ # The video in the original webpage now uses PlayWire
+ 'skip': 'Ooyala said: movie expired',
}, {
# Only available for ipad
'url': 'http://player.ooyala.com/player.js?embedCode=x1b3lqZDq9y_7kMyC2Op5qo-p077tXD0',
from .common import InfoExtractor
from ..compat import compat_chr
from ..utils import (
+ determine_ext,
encode_base_n,
ExtractorError,
+ mimetype2ext,
)
}, {
'url': 'https://openload.io/f/ZAn6oz-VZGE/',
'only_matching': True,
+ }, {
+ # unavailable via https://openload.co/f/Sxz5sADo82g/, different layout
+ # for title and ext
+ 'url': 'https://openload.co/embed/Sxz5sADo82g/',
+ 'only_matching': True,
}]
@staticmethod
raise ExtractorError('File not found', expected=True)
code = self._search_regex(
- r'<video[^>]+>\s*<script[^>]+>([^<]+)</script>',
+ r'</video>\s*</div>\s*<script[^>]+>([^<]+)</script>',
webpage, 'JS code')
+ decoded = self.openload_decode(code)
+
video_url = self._search_regex(
- r'return\s+"(https?://[^"]+)"', self.openload_decode(code), 'video URL')
+ r'return\s+"(https?://[^"]+)"', decoded, 'video URL')
+
+ title = self._og_search_title(webpage, default=None) or self._search_regex(
+ r'<span[^>]+class=["\']title["\'][^>]*>([^<]+)', webpage,
+ 'title', default=None) or self._html_search_meta(
+ 'description', webpage, 'title', fatal=True)
+
+ ext = mimetype2ext(self._search_regex(
+ r'window\.vt\s*=\s*(["\'])(?P<mimetype>.+?)\1', decoded,
+ 'mimetype', default=None, group='mimetype')) or determine_ext(
+ video_url, 'mp4')
return {
'id': video_id,
- 'title': self._og_search_title(webpage),
- 'thumbnail': self._og_search_thumbnail(webpage),
+ 'title': title,
+ 'ext': ext,
+ 'thumbnail': self._og_search_thumbnail(webpage, default=None),
'url': video_url,
}
class OraTVIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?ora\.tv/([^/]+/)*(?P<id>[^/\?#]+)'
- _TEST = {
+ _VALID_URL = r'https?://(?:www\.)?(?:ora\.tv|unsafespeech\.com)/([^/]+/)*(?P<id>[^/\?#]+)'
+ _TESTS = [{
'url': 'https://www.ora.tv/larrykingnow/2015/12/16/vine-youtube-stars-zach-king-king-bach-on-their-viral-videos-0_36jupg6090pq',
'md5': 'fa33717591c631ec93b04b0e330df786',
'info_dict': {
'title': 'Vine & YouTube Stars Zach King & King Bach On Their Viral Videos!',
'description': 'md5:ebbc5b1424dd5dba7be7538148287ac1',
}
- }
+ }, {
+ 'url': 'http://www.unsafespeech.com/video/2016/5/10/student-self-censorship-and-the-thought-police-on-university-campuses-0_6622bnkppw4d',
+ 'only_matching': True,
+ }]
def _real_extract(self, url):
display_id = self._match_id(url)
'timestamp': 1452456073,
'upload_date': '20160110',
},
+ 'skip': 'Live streams on FM4 got deleted soon',
}
def _real_extract(self, url):
_TESTS = [
{
'url': 'http://www.pbs.org/tpt/constitution-usa-peter-sagal/watch/a-more-perfect-union/',
- 'md5': 'ce1888486f0908d555a8093cac9a7362',
+ 'md5': '173dc391afd361fa72eab5d3d918968d',
'info_dict': {
'id': '2365006249',
'ext': 'mp4',
'description': 'md5:36f341ae62e251b8f5bd2b754b95a071',
'duration': 3190,
},
- 'params': {
- 'skip_download': True, # requires ffmpeg
- },
},
{
'url': 'http://www.pbs.org/wgbh/pages/frontline/losing-iraq/',
- 'md5': '143c98aa54a346738a3d78f54c925321',
+ 'md5': '6f722cb3c3982186d34b0f13374499c7',
'info_dict': {
'id': '2365297690',
'ext': 'mp4',
'description': 'md5:4d3eaa01f94e61b3e73704735f1196d9',
'duration': 5050,
},
- 'params': {
- 'skip_download': True, # requires ffmpeg
- }
},
{
'url': 'http://www.pbs.org/newshour/bb/education-jan-june12-cyberschools_02-23/',
'duration': 6559,
'thumbnail': 're:^https?://.*\.jpg$',
},
- 'params': {
- 'skip_download': True, # requires ffmpeg
- },
},
{
'url': 'http://www.pbs.org/wgbh/nova/earth/killer-typhoon.html',
'upload_date': '20140122',
'age_limit': 10,
},
- 'params': {
- 'skip_download': True, # requires ffmpeg
- },
},
{
'url': 'http://www.pbs.org/wgbh/pages/frontline/united-states-of-secrets/',
},
{
'url': 'http://www.pbs.org/video/2365245528/',
+ 'md5': '115223d41bd55cda8ae5cd5ed4e11497',
'info_dict': {
'id': '2365245528',
'display_id': '2365245528',
'duration': 6851,
'thumbnail': 're:^https?://.*\.jpg$',
},
- 'params': {
- 'skip_download': True, # requires ffmpeg
- },
},
{
# Video embedded in iframe containing angle brackets as attribute's value (e.g.
# "<iframe style='position: absolute;<br />\ntop: 0; left: 0;' ...", see
# https://github.com/rg3/youtube-dl/issues/7059)
'url': 'http://www.pbs.org/food/features/a-chefs-life-season-3-episode-5-prickly-business/',
+ 'md5': '84ced42850d78f1d4650297356e95e6f',
'info_dict': {
'id': '2365546844',
'display_id': 'a-chefs-life-season-3-episode-5-prickly-business',
'duration': 1480,
'thumbnail': 're:^https?://.*\.jpg$',
},
- 'params': {
- 'skip_download': True, # requires ffmpeg
- },
},
{
# Frontline video embedded via flp2012.js
{
# Serves hd only via wigget/partnerplayer page
'url': 'http://www.pbs.org/video/2365641075/',
+ 'md5': 'acfd4c400b48149a44861cb16dd305cf',
'info_dict': {
'id': '2365641075',
'ext': 'mp4',
'thumbnail': 're:^https?://.*\.jpg$',
'formats': 'mincount:8',
},
- 'params': {
- 'skip_download': True, # requires ffmpeg
- },
},
{
'url': 'http://player.pbs.org/widget/partnerplayer/2365297708/?start=0&end=0&chapterbar=false&endscreen=false&topbar=true',
info = video_info
formats = []
+ http_url = None
for num, redirect in enumerate(redirects):
redirect_id = redirect.get('eeid')
if determine_ext(format_url) == 'm3u8':
formats.extend(self._extract_m3u8_formats(
- format_url, display_id, 'mp4', preference=1, m3u8_id='hls'))
+ format_url, display_id, 'mp4', m3u8_id='hls', fatal=False))
else:
formats.append({
'url': format_url,
'format_id': redirect_id,
})
+ if re.search(r'^https?://.*(?:\d+k|baseline)', format_url):
+ http_url = format_url
self._remove_duplicate_formats(formats)
+ m3u8_formats = list(filter(
+ lambda f: f.get('protocol') == 'm3u8' and f.get('vcodec') != 'none' and f.get('resolution') != 'multiple',
+ formats))
+ if http_url:
+ for m3u8_format in m3u8_formats:
+ bitrate = self._search_regex(r'(\d+k)', m3u8_format['url'], 'bitrate', default=None)
+ # extract only the formats that we know that they will be available as http format.
+ # https://projects.pbs.org/confluence/display/coveapi/COVE+Video+Specifications
+ if not bitrate or bitrate not in ('400k', '800k', '1200k', '2500k'):
+ continue
+ f = m3u8_format.copy()
+ f.update({
+ 'url': re.sub(r'\d+k|baseline', bitrate, http_url),
+ 'format_id': m3u8_format['format_id'].replace('hls', 'http'),
+ 'protocol': 'http',
+ })
+ formats.append(f)
self._sort_formats(formats)
rating_str = info.get('rating')
'ext': 'ttml',
'url': closed_captions_url,
}]
+ mobj = re.search(r'/(\d+)_Encoded\.dfxp', closed_captions_url)
+ if mobj:
+ ttml_caption_suffix, ttml_caption_id = mobj.group(0, 1)
+ ttml_caption_id = int(ttml_caption_id)
+ subtitles['en'].extend([{
+ 'url': closed_captions_url.replace(
+ ttml_caption_suffix, '/%d_Encoded.srt' % (ttml_caption_id + 1)),
+ 'ext': 'srt',
+ }, {
+ 'url': closed_captions_url.replace(
+ ttml_caption_suffix, '/%d_Encoded.vtt' % (ttml_caption_id + 2)),
+ 'ext': 'vtt',
+ }])
# info['title'] is often incomplete (e.g. 'Full Episode', 'Episode 5', etc)
# Try turning it to 'program - title' naming scheme if possible
from __future__ import unicode_literals
from .common import InfoExtractor
-from ..utils import parse_iso8601
+from ..utils import (
+ parse_iso8601,
+ unescapeHTML,
+)
class PeriscopeIE(InfoExtractor):
IE_DESC = 'Periscope'
+ IE_NAME = 'periscope'
_VALID_URL = r'https?://(?:www\.)?periscope\.tv/[^/]+/(?P<id>[^/?#]+)'
# Alive example URLs can be found here http://onperiscope.com/
_TESTS = [{
broadcast = broadcast_data['broadcast']
status = broadcast['status']
- uploader = broadcast.get('user_display_name') or broadcast_data.get('user', {}).get('display_name')
- uploader_id = broadcast.get('user_id') or broadcast_data.get('user', {}).get('id')
+ user = broadcast_data.get('user', {})
+
+ uploader = broadcast.get('user_display_name') or user.get('display_name')
+ uploader_id = (broadcast.get('username') or user.get('username') or
+ broadcast.get('user_id') or user.get('id'))
title = '%s - %s' % (uploader, status) if uploader else status
state = broadcast.get('state').lower()
'thumbnails': thumbnails,
'formats': formats,
}
+
+
+class PeriscopeUserIE(InfoExtractor):
+ _VALID_URL = r'https?://www\.periscope\.tv/(?P<id>[^/]+)/?$'
+ IE_DESC = 'Periscope user videos'
+ IE_NAME = 'periscope:user'
+
+ _TEST = {
+ 'url': 'https://www.periscope.tv/LularoeHusbandMike/',
+ 'info_dict': {
+ 'id': 'LularoeHusbandMike',
+ 'title': 'LULAROE HUSBAND MIKE',
+ 'description': 'md5:6cf4ec8047768098da58e446e82c82f0',
+ },
+ # Periscope only shows videos in the last 24 hours, so it's possible to
+ # get 0 videos
+ 'playlist_mincount': 0,
+ }
+
+ def _real_extract(self, url):
+ user_id = self._match_id(url)
+
+ webpage = self._download_webpage(url, user_id)
+
+ data_store = self._parse_json(
+ unescapeHTML(self._search_regex(
+ r'data-store=(["\'])(?P<data>.+?)\1',
+ webpage, 'data store', default='{}', group='data')),
+ user_id)
+
+ user = data_store.get('User', {}).get('user', {})
+ title = user.get('display_name') or user.get('username')
+ description = user.get('description')
+
+ entries = [
+ self.url_result(
+ 'https://www.periscope.tv/%s/%s' % (user_id, broadcast['id']))
+ for broadcast in data_store.get('UserBroadcastHistory', {}).get('broadcasts', [])]
+
+ return self.playlist_result(entries, user_id, title, description)
from .common import InfoExtractor
from ..utils import (
- xpath_text,
+ dict_get,
float_or_none,
- int_or_none,
)
'duration': 145.94,
},
}, {
+ # m3u8 in f4m
+ 'url': 'http://config.playwire.com/21772/videos/v2/4840492/zeus.json',
+ 'info_dict': {
+ 'id': '4840492',
+ 'ext': 'mp4',
+ 'title': 'ITV EL SHOW FULL',
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
+ }, {
+ # Multiple resolutions while bitrates missing
'url': 'http://cdn.playwire.com/11625/embed/85228.html',
'only_matching': True,
}, {
thumbnail = content.get('poster')
src = content['media']['f4m']
- f4m = self._download_xml(src, video_id)
- base_url = xpath_text(f4m, './{http://ns.adobe.com/f4m/1.0}baseURL', 'base url', fatal=True)
- formats = []
- for media in f4m.findall('./{http://ns.adobe.com/f4m/1.0}media'):
- media_url = media.get('url')
- if not media_url:
- continue
- tbr = int_or_none(media.get('bitrate'))
- width = int_or_none(media.get('width'))
- height = int_or_none(media.get('height'))
- f = {
- 'url': '%s/%s' % (base_url, media.attrib['url']),
- 'tbr': tbr,
- 'width': width,
- 'height': height,
- }
- if not (tbr or width or height):
- f['quality'] = 1 if '-hd.' in media_url else 0
- formats.append(f)
+ formats = self._extract_f4m_formats(src, video_id, m3u8_id='hls')
+ for a_format in formats:
+ if not dict_get(a_format, ['tbr', 'width', 'height']):
+ a_format['quality'] = 1 if '-hd.' in a_format['url'] else 0
self._sort_formats(formats)
return {
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+ xpath_text,
+ find_xpath_attr,
+ determine_ext,
+ int_or_none,
+ unified_strdate,
+ xpath_element,
+ ExtractorError,
+)
+
+
+class RadioCanadaIE(InfoExtractor):
+ IE_NAME = 'radiocanada'
+ _VALID_URL = r'(?:radiocanada:|https?://ici\.radio-canada\.ca/widgets/mediaconsole/)(?P<app_code>[^:/]+)[:/](?P<id>[0-9]+)'
+ _TEST = {
+ 'url': 'http://ici.radio-canada.ca/widgets/mediaconsole/medianet/7184272',
+ 'info_dict': {
+ 'id': '7184272',
+ 'ext': 'flv',
+ 'title': 'Le parcours du tireur capté sur vidéo',
+ 'description': 'Images des caméras de surveillance fournies par la GRC montrant le parcours du tireur d\'Ottawa',
+ 'upload_date': '20141023',
+ },
+ 'params': {
+ # rtmp download
+ 'skip_download': True,
+ },
+ }
+
+ def _real_extract(self, url):
+ app_code, video_id = re.match(self._VALID_URL, url).groups()
+
+ formats = []
+ # TODO: extract m3u8 and f4m formats
+ # m3u8 formats can be extracted using ipad device_type return 403 error code when ffmpeg try to download segements
+ # f4m formats can be extracted using flashhd device_type but they produce unplayable file
+ for device_type in ('flash',):
+ v_data = self._download_xml(
+ 'http://api.radio-canada.ca/validationMedia/v1/Validation.ashx',
+ video_id, note='Downloading %s XML' % device_type, query={
+ 'appCode': app_code,
+ 'idMedia': video_id,
+ 'connectionType': 'broadband',
+ 'multibitrate': 'true',
+ 'deviceType': device_type,
+ # paysJ391wsHjbOJwvCs26toz and bypasslock are used to bypass geo-restriction
+ 'paysJ391wsHjbOJwvCs26toz': 'CA',
+ 'bypasslock': 'NZt5K62gRqfc',
+ })
+ v_url = xpath_text(v_data, 'url')
+ if not v_url:
+ continue
+ if v_url == 'null':
+ raise ExtractorError('%s said: %s' % (
+ self.IE_NAME, xpath_text(v_data, 'message')), expected=True)
+ ext = determine_ext(v_url)
+ if ext == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ v_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+ elif ext == 'f4m':
+ formats.extend(self._extract_f4m_formats(v_url, video_id, f4m_id='hds', fatal=False))
+ else:
+ ext = determine_ext(v_url)
+ bitrates = xpath_element(v_data, 'bitrates')
+ for url_e in bitrates.findall('url'):
+ tbr = int_or_none(url_e.get('bitrate'))
+ if not tbr:
+ continue
+ formats.append({
+ 'format_id': 'rtmp-%d' % tbr,
+ 'url': re.sub(r'\d+\.%s' % ext, '%d.%s' % (tbr, ext), v_url),
+ 'ext': 'flv',
+ 'protocol': 'rtmp',
+ 'width': int_or_none(url_e.get('width')),
+ 'height': int_or_none(url_e.get('height')),
+ 'tbr': tbr,
+ })
+ self._sort_formats(formats)
+
+ metadata = self._download_xml(
+ 'http://api.radio-canada.ca/metaMedia/v1/index.ashx',
+ video_id, note='Downloading metadata XML', query={
+ 'appCode': app_code,
+ 'idMedia': video_id,
+ })
+
+ def get_meta(name):
+ el = find_xpath_attr(metadata, './/Meta', 'name', name)
+ return el.text if el is not None else None
+
+ return {
+ 'id': video_id,
+ 'title': get_meta('Title'),
+ 'description': get_meta('Description') or get_meta('ShortDescription'),
+ 'thumbnail': get_meta('imageHR') or get_meta('imageMR') or get_meta('imageBR'),
+ 'duration': int_or_none(get_meta('length')),
+ 'series': get_meta('Emission'),
+ 'season_number': int_or_none('SrcSaison'),
+ 'episode_number': int_or_none('SrcEpisode'),
+ 'upload_date': unified_strdate(get_meta('Date')),
+ 'formats': formats,
+ }
+
+
+class RadioCanadaAudioVideoIE(InfoExtractor):
+ 'radiocanada:audiovideo'
+ _VALID_URL = r'https?://ici\.radio-canada\.ca/audio-video/media-(?P<id>[0-9]+)'
+ _TEST = {
+ 'url': 'http://ici.radio-canada.ca/audio-video/media-7527184/barack-obama-au-vietnam',
+ 'info_dict': {
+ 'id': '7527184',
+ 'ext': 'flv',
+ 'title': 'Barack Obama au Vietnam',
+ 'description': 'Les États-Unis lèvent l\'embargo sur la vente d\'armes qui datait de la guerre du Vietnam',
+ 'upload_date': '20160523',
+ },
+ 'params': {
+ # rtmp download
+ 'skip_download': True,
+ },
+ }
+
+ def _real_extract(self, url):
+ return self.url_result('radiocanada:medianet:%s' % self._match_id(url))
from __future__ import unicode_literals
from .common import InfoExtractor
-from ..utils import ExtractorError
+from ..utils import (
+ ExtractorError,
+ int_or_none,
+ str_to_int,
+ unified_strdate,
+)
class RedTubeIE(InfoExtractor):
'id': '66418',
'ext': 'mp4',
'title': 'Sucked on a toilet',
+ 'upload_date': '20120831',
+ 'duration': 596,
+ 'view_count': int,
'age_limit': 18,
}
}
if any(s in webpage for s in ['video-deleted-info', '>This video has been removed']):
raise ExtractorError('Video %s has been removed' % video_id, expected=True)
- video_url = self._html_search_regex(
- r'<source src="(.+?)" type="video/mp4">', webpage, 'video URL')
- video_title = self._html_search_regex(
- r'<h1 class="videoTitle[^"]*">(.+?)</h1>',
- webpage, 'title')
- video_thumbnail = self._og_search_thumbnail(webpage)
+ title = self._html_search_regex(
+ (r'<h1 class="videoTitle[^"]*">(?P<title>.+?)</h1>',
+ r'videoTitle\s*:\s*(["\'])(?P<title>)\1'),
+ webpage, 'title', group='title')
+
+ formats = []
+ sources = self._parse_json(
+ self._search_regex(
+ r'sources\s*:\s*({.+?})', webpage, 'source', default='{}'),
+ video_id, fatal=False)
+ if sources and isinstance(sources, dict):
+ for format_id, format_url in sources.items():
+ if format_url:
+ formats.append({
+ 'url': format_url,
+ 'format_id': format_id,
+ 'height': int_or_none(format_id),
+ })
+ else:
+ video_url = self._html_search_regex(
+ r'<source src="(.+?)" type="video/mp4">', webpage, 'video URL')
+ formats.append({'url': video_url})
+ self._sort_formats(formats)
+
+ thumbnail = self._og_search_thumbnail(webpage)
+ upload_date = unified_strdate(self._search_regex(
+ r'<span[^>]+class="added-time"[^>]*>ADDED ([^<]+)<',
+ webpage, 'upload date', fatal=False))
+ duration = int_or_none(self._search_regex(
+ r'videoDuration\s*:\s*(\d+)', webpage, 'duration', fatal=False))
+ view_count = str_to_int(self._search_regex(
+ r'<span[^>]*>VIEWS</span></td>\s*<td>([\d,.]+)',
+ webpage, 'view count', fatal=False))
# No self-labeling, but they describe themselves as
# "Home of Videos Porno"
return {
'id': video_id,
- 'url': video_url,
'ext': 'mp4',
- 'title': video_title,
- 'thumbnail': video_thumbnail,
+ 'title': title,
+ 'thumbnail': thumbnail,
+ 'upload_date': upload_date,
+ 'duration': duration,
+ 'view_count': view_count,
'age_limit': age_limit,
+ 'formats': formats,
}
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+ js_to_json,
+ int_or_none,
+ unescapeHTML,
+)
+
+
+class ReutersIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?reuters\.com/.*?\?.*?videoId=(?P<id>[0-9]+)'
+ _TEST = {
+ 'url': 'http://www.reuters.com/video/2016/05/20/san-francisco-police-chief-resigns?videoId=368575562',
+ 'md5': '8015113643a0b12838f160b0b81cc2ee',
+ 'info_dict': {
+ 'id': '368575562',
+ 'ext': 'mp4',
+ 'title': 'San Francisco police chief resigns',
+ }
+ }
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ webpage = self._download_webpage(
+ 'http://www.reuters.com/assets/iframe/yovideo?videoId=%s' % video_id, video_id)
+ video_data = js_to_json(self._search_regex(
+ r'(?s)Reuters\.yovideo\.drawPlayer\(({.*?})\);',
+ webpage, 'video data'))
+
+ def get_json_value(key, fatal=False):
+ return self._search_regex('"%s"\s*:\s*"([^"]+)"' % key, video_data, key, fatal=fatal)
+
+ title = unescapeHTML(get_json_value('title', fatal=True))
+ mmid, fid = re.search(r',/(\d+)\?f=(\d+)', get_json_value('flv', fatal=True)).groups()
+
+ mas_data = self._download_json(
+ 'http://mas-e.cds1.yospace.com/mas/%s/%s?trans=json' % (mmid, fid),
+ video_id, transform_source=js_to_json)
+ formats = []
+ for f in mas_data:
+ f_url = f.get('url')
+ if not f_url:
+ continue
+ method = f.get('method')
+ if method == 'hls':
+ formats.extend(self._extract_m3u8_formats(
+ f_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
+ else:
+ container = f.get('container')
+ ext = '3gp' if method == 'mobile' else container
+ formats.append({
+ 'format_id': ext,
+ 'url': f_url,
+ 'ext': ext,
+ 'container': container if method != 'mobile' else None,
+ })
+ self._sort_formats(formats)
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'thumbnail': get_json_value('thumb'),
+ 'duration': int_or_none(get_json_value('seconds')),
+ 'formats': formats,
+ }
)
+class Revision3EmbedIE(InfoExtractor):
+ IE_NAME = 'revision3:embed'
+ _VALID_URL = r'(?:revision3:(?:(?P<playlist_type>[^:]+):)?|https?://(?:(?:(?:www|embed)\.)?(?:revision3|animalist)|(?:(?:api|embed)\.)?seekernetwork)\.com/player/embed\?videoId=)(?P<playlist_id>\d+)'
+ _TEST = {
+ 'url': 'http://api.seekernetwork.com/player/embed?videoId=67558',
+ 'md5': '83bcd157cab89ad7318dd7b8c9cf1306',
+ 'info_dict': {
+ 'id': '67558',
+ 'ext': 'mp4',
+ 'title': 'The Pros & Cons Of Zoos',
+ 'description': 'Zoos are often depicted as a terrible place for animals to live, but is there any truth to this?',
+ 'uploader_id': 'dnews',
+ 'uploader': 'DNews',
+ }
+ }
+ _API_KEY = 'ba9c741bce1b9d8e3defcc22193f3651b8867e62'
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ playlist_id = mobj.group('playlist_id')
+ playlist_type = mobj.group('playlist_type') or 'video_id'
+ video_data = self._download_json(
+ 'http://revision3.com/api/getPlaylist.json', playlist_id, query={
+ 'api_key': self._API_KEY,
+ 'codecs': 'h264,vp8,theora',
+ playlist_type: playlist_id,
+ })['items'][0]
+
+ formats = []
+ for vcodec, media in video_data['media'].items():
+ for quality_id, quality in media.items():
+ if quality_id == 'hls':
+ formats.extend(self._extract_m3u8_formats(
+ quality['url'], playlist_id, 'mp4',
+ 'm3u8_native', m3u8_id='hls', fatal=False))
+ else:
+ formats.append({
+ 'url': quality['url'],
+ 'format_id': '%s-%s' % (vcodec, quality_id),
+ 'tbr': int_or_none(quality.get('bitrate')),
+ 'vcodec': vcodec,
+ })
+ self._sort_formats(formats)
+
+ return {
+ 'id': playlist_id,
+ 'title': unescapeHTML(video_data['title']),
+ 'description': unescapeHTML(video_data.get('summary')),
+ 'uploader': video_data.get('show', {}).get('name'),
+ 'uploader_id': video_data.get('show', {}).get('slug'),
+ 'duration': int_or_none(video_data.get('duration')),
+ 'formats': formats,
+ }
+
+
class Revision3IE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?(?P<domain>(?:revision3|testtube|animalist)\.com)/(?P<id>[^/]+(?:/[^/?#]+)?)'
+ IE_NAME = 'revision'
+ _VALID_URL = r'https?://(?:www\.)?(?P<domain>(?:revision3|animalist)\.com)/(?P<id>[^/]+(?:/[^/?#]+)?)'
_TESTS = [{
'url': 'http://www.revision3.com/technobuffalo/5-google-predictions-for-2016',
'md5': 'd94a72d85d0a829766de4deb8daaf7df',
}
}, {
# Show
- 'url': 'http://testtube.com/brainstuff',
- 'info_dict': {
- 'id': '251',
- 'title': 'BrainStuff',
- 'description': 'Whether the topic is popcorn or particle physics, you can count on the HowStuffWorks team to explore-and explain-the everyday science in the world around us on BrainStuff.',
- },
- 'playlist_mincount': 93,
- }, {
- 'url': 'https://testtube.com/dnews/5-weird-ways-plants-can-eat-animals?utm_source=FB&utm_medium=DNews&utm_campaign=DNewsSocial',
- 'info_dict': {
- 'id': '58227',
- 'display_id': 'dnews/5-weird-ways-plants-can-eat-animals',
- 'duration': 275,
- 'ext': 'webm',
- 'title': '5 Weird Ways Plants Can Eat Animals',
- 'description': 'Why have some plants evolved to eat meat?',
- 'upload_date': '20150120',
- 'timestamp': 1421763300,
- 'uploader': 'DNews',
- 'uploader_id': 'dnews',
- },
- }, {
- 'url': 'http://testtube.com/tt-editors-picks/the-israel-palestine-conflict-explained-in-ten-min',
- 'info_dict': {
- 'id': '71618',
- 'ext': 'mp4',
- 'display_id': 'tt-editors-picks/the-israel-palestine-conflict-explained-in-ten-min',
- 'title': 'The Israel-Palestine Conflict Explained in Ten Minutes',
- 'description': 'If you\'d like to learn about the struggle between Israelis and Palestinians, this video is a great place to start',
- 'uploader': 'Editors\' Picks',
- 'uploader_id': 'tt-editors-picks',
- 'timestamp': 1453309200,
- 'upload_date': '20160120',
- },
- 'add_ie': ['Youtube'],
+ 'url': 'http://revision3.com/variant',
+ 'only_matching': True,
}, {
# Tag
- 'url': 'http://testtube.com/tech-news',
- 'info_dict': {
- 'id': '21018',
- 'title': 'tech news',
- },
- 'playlist_mincount': 9,
+ 'url': 'http://revision3.com/vr',
+ 'only_matching': True,
}]
_PAGE_DATA_TEMPLATE = 'http://www.%s/apiProxy/ddn/%s?domain=%s'
- _API_KEY = 'ba9c741bce1b9d8e3defcc22193f3651b8867e62'
def _real_extract(self, url):
domain, display_id = re.match(self._VALID_URL, url).groups()
})
return info
- video_data = self._download_json(
- 'http://revision3.com/api/getPlaylist.json?api_key=%s&codecs=h264,vp8,theora&video_id=%s' % (self._API_KEY, video_id),
- video_id)['items'][0]
-
- formats = []
- for vcodec, media in video_data['media'].items():
- for quality_id, quality in media.items():
- if quality_id == 'hls':
- formats.extend(self._extract_m3u8_formats(
- quality['url'], video_id, 'mp4',
- 'm3u8_native', m3u8_id='hls', fatal=False))
- else:
- formats.append({
- 'url': quality['url'],
- 'format_id': '%s-%s' % (vcodec, quality_id),
- 'tbr': int_or_none(quality.get('bitrate')),
- 'vcodec': vcodec,
- })
- self._sort_formats(formats)
-
info.update({
- 'title': unescapeHTML(video_data['title']),
- 'description': unescapeHTML(video_data.get('summary')),
- 'uploader': video_data.get('show', {}).get('name'),
- 'uploader_id': video_data.get('show', {}).get('slug'),
- 'duration': int_or_none(video_data.get('duration')),
- 'formats': formats,
+ '_type': 'url_transparent',
+ 'url': 'revision3:%s' % video_id,
})
return info
else:
(?P<id>[0-9a-f-]+)'''
_TESTS = [{
- 'url': 'http://www.rtlxl.nl/#!/rtl-nieuws-132237/6e4203a6-0a5e-3596-8424-c599a59e0677',
- 'md5': 'cc16baa36a6c169391f0764fa6b16654',
+ 'url': 'http://www.rtlxl.nl/#!/rtl-nieuws-132237/82b1aad1-4a14-3d7b-b554-b0aed1b2c416',
+ 'md5': '473d1946c1fdd050b2c0161a4b13c373',
'info_dict': {
- 'id': '6e4203a6-0a5e-3596-8424-c599a59e0677',
+ 'id': '82b1aad1-4a14-3d7b-b554-b0aed1b2c416',
'ext': 'mp4',
- 'title': 'RTL Nieuws - Laat',
- 'description': 'md5:6b61f66510c8889923b11f2778c72dc5',
- 'timestamp': 1408051800,
- 'upload_date': '20140814',
- 'duration': 576.880,
+ 'title': 'RTL Nieuws',
+ 'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
+ 'timestamp': 1461951000,
+ 'upload_date': '20160429',
+ 'duration': 1167.96,
},
}, {
+ # best format avaialble a3t
'url': 'http://www.rtl.nl/system/videoplayer/derden/rtlnieuws/video_embed.html#uuid=84ae5571-ac25-4225-ae0c-ef8d9efb2aed/autoplay=false',
'md5': 'dea7474214af1271d91ef332fb8be7ea',
'info_dict': {
'ext': 'mp4',
'timestamp': 1424039400,
'title': 'RTL Nieuws - Nieuwe beelden Kopenhagen: chaos direct na aanslag',
- 'thumbnail': 're:^https?://screenshots\.rtl\.nl/system/thumb/sz=[0-9]+x[0-9]+/uuid=84ae5571-ac25-4225-ae0c-ef8d9efb2aed$',
+ 'thumbnail': 're:^https?://screenshots\.rtl\.nl/(?:[^/]+/)*sz=[0-9]+x[0-9]+/uuid=84ae5571-ac25-4225-ae0c-ef8d9efb2aed$',
'upload_date': '20150215',
'description': 'Er zijn nieuwe beelden vrijgegeven die vlak na de aanslag in Kopenhagen zijn gemaakt. Op de video is goed te zien hoe omstanders zich bekommeren om één van de slachtoffers, terwijl de eerste agenten ter plaatse komen.',
}
}, {
# empty synopsis and missing episodes (see https://github.com/rg3/youtube-dl/issues/6275)
+ # best format available nettv
'url': 'http://www.rtl.nl/system/videoplayer/derden/rtlnieuws/video_embed.html#uuid=f536aac0-1dc3-4314-920e-3bd1c5b3811a/autoplay=false',
'info_dict': {
'id': 'f536aac0-1dc3-4314-920e-3bd1c5b3811a',
'ext': 'mp4',
'title': 'RTL Nieuws - Meer beelden van overval juwelier',
- 'thumbnail': 're:^https?://screenshots\.rtl\.nl/system/thumb/sz=[0-9]+x[0-9]+/uuid=f536aac0-1dc3-4314-920e-3bd1c5b3811a$',
+ 'thumbnail': 're:^https?://screenshots\.rtl\.nl/(?:[^/]+/)*sz=[0-9]+x[0-9]+/uuid=f536aac0-1dc3-4314-920e-3bd1c5b3811a$',
'timestamp': 1437233400,
'upload_date': '20150718',
'duration': 30.474,
videopath = material['videopath']
m3u8_url = meta.get('videohost', 'http://manifest.us.rtl.nl') + videopath
- formats = self._extract_m3u8_formats(m3u8_url, uuid, ext='mp4')
+ formats = self._extract_m3u8_formats(
+ m3u8_url, uuid, 'mp4', m3u8_id='hls', fatal=False)
video_urlpart = videopath.split('/adaptive/')[1][:-5]
PG_URL_TEMPLATE = 'http://pg.us.rtl.nl/rtlxl/network/%s/progressive/%s.mp4'
- formats.extend([
- {
- 'url': PG_URL_TEMPLATE % ('a2m', video_urlpart),
- 'format_id': 'pg-sd',
- },
- {
- 'url': PG_URL_TEMPLATE % ('a3m', video_urlpart),
- 'format_id': 'pg-hd',
- 'quality': 0,
+ PG_FORMATS = (
+ ('a2t', 512, 288),
+ ('a3t', 704, 400),
+ ('nettv', 1280, 720),
+ )
+
+ def pg_format(format_id, width, height):
+ return {
+ 'url': PG_URL_TEMPLATE % (format_id, video_urlpart),
+ 'format_id': 'pg-%s' % format_id,
+ 'protocol': 'http',
+ 'width': width,
+ 'height': height,
}
- ])
+
+ if not formats:
+ formats = [pg_format(*pg_tuple) for pg_tuple in PG_FORMATS]
+ else:
+ pg_formats = []
+ for format_id, width, height in PG_FORMATS:
+ try:
+ # Find hls format with the same width and height corresponding
+ # to progressive format and copy metadata from it.
+ f = next(f for f in formats if f.get('height') == height)
+ # hls formats may have invalid width
+ f['width'] = width
+ f_copy = f.copy()
+ f_copy.update(pg_format(format_id, width, height))
+ pg_formats.append(f_copy)
+ except StopIteration:
+ # Missing hls format does mean that no progressive format with
+ # such width and height exists either.
+ pass
+ formats.extend(pg_formats)
self._sort_formats(formats)
thumbnails = []
import time
from .common import InfoExtractor
+from ..compat import (
+ compat_struct_unpack,
+)
from ..utils import (
ExtractorError,
float_or_none,
remove_start,
sanitized_Request,
std_headers,
- struct_unpack,
)
encrypted_data = base64.b64decode(png.encode('utf-8'))
text_index = encrypted_data.find(b'tEXt')
text_chunk = encrypted_data[text_index - 4:]
- length = struct_unpack('!I', text_chunk[:4])[0]
+ length = compat_struct_unpack('!I', text_chunk[:4])[0]
# Use bytearray to get integers when iterating in both python 2.x and 3.x
data = bytearray(text_chunk[8:8 + length])
data = [chr(b) for b in data if b != 0]
class RTVEALaCartaIE(InfoExtractor):
IE_NAME = 'rtve.es:alacarta'
IE_DESC = 'RTVE a la carta'
- _VALID_URL = r'https?://www\.rtve\.es/(m/)?alacarta/videos/[^/]+/[^/]+/(?P<id>\d+)'
+ _VALID_URL = r'https?://www\.rtve\.es/(m/)?(alacarta/videos|filmoteca)/[^/]+/[^/]+/(?P<id>\d+)'
_TESTS = [{
'url': 'http://www.rtve.es/alacarta/videos/balonmano/o-swiss-cup-masculina-final-espana-suecia/2491869/',
}, {
'url': 'http://www.rtve.es/m/alacarta/videos/cuentame-como-paso/cuentame-como-paso-t16-ultimo-minuto-nuestra-vida-capitulo-276/2969138/?media=tve',
'only_matching': True,
+ }, {
+ 'url': 'http://www.rtve.es/filmoteca/no-do/not-1-introduccion-primer-noticiario-espanol/1465256/',
+ 'only_matching': True,
}]
def _real_initialize(self):
'title': 'Adam Arkin at the 2014 DOE JGI Genomics of Energy & Environment Meeting',
'description': 'md5:81f1710638e11a481358fab1b11059d7',
},
+ 'skip': 'Not accessible from Travis CI server',
}
def _real_extract(self, url):
class ScreenwaveMediaIE(InfoExtractor):
- _VALID_URL = r'https?://player\d?\.screenwavemedia\.com/(?:play/)?[a-zA-Z]+\.php\?.*\bid=(?P<id>[A-Za-z0-9-]+)'
+ _VALID_URL = r'(?:https?:)?//player\d?\.screenwavemedia\.com/(?:play/)?[a-zA-Z]+\.php\?.*\bid=(?P<id>[A-Za-z0-9-]+)'
EMBED_PATTERN = r'src=(["\'])(?P<url>(?:https?:)?//player\d?\.screenwavemedia\.com/(?:play/)?[a-zA-Z]+\.php\?.*\bid=.+?)\1'
_TESTS = [{
'url': 'http://player.screenwavemedia.com/play/play.php?playerdiv=videoarea&companiondiv=squareAd&id=Cinemassacre-19911',
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+
+
+class SeekerIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?seeker\.com/(?P<display_id>.*)-(?P<article_id>\d+)\.html'
+ _TESTS = [{
+ # player.loadRevision3Item
+ 'url': 'http://www.seeker.com/should-trump-be-required-to-release-his-tax-returns-1833805621.html',
+ 'md5': '30c1dc4030cc715cf05b423d0947ac18',
+ 'info_dict': {
+ 'id': '76243',
+ 'ext': 'webm',
+ 'title': 'Should Trump Be Required To Release His Tax Returns?',
+ 'description': 'Donald Trump has been secretive about his "big," "beautiful" tax returns. So what can we learn if he decides to release them?',
+ 'uploader': 'Seeker Daily',
+ 'uploader_id': 'seekerdaily',
+ }
+ }, {
+ 'url': 'http://www.seeker.com/changes-expected-at-zoos-following-recent-gorilla-lion-shootings-1834116536.html',
+ 'playlist': [
+ {
+ 'md5': '83bcd157cab89ad7318dd7b8c9cf1306',
+ 'info_dict': {
+ 'id': '67558',
+ 'ext': 'mp4',
+ 'title': 'The Pros & Cons Of Zoos',
+ 'description': 'Zoos are often depicted as a terrible place for animals to live, but is there any truth to this?',
+ 'uploader': 'DNews',
+ 'uploader_id': 'dnews',
+ },
+ }
+ ],
+ 'info_dict': {
+ 'id': '1834116536',
+ 'title': 'After Gorilla Killing, Changes Ahead for Zoos',
+ 'description': 'The largest association of zoos and others are hoping to learn from recent incidents that led to the shooting deaths of a gorilla and two lions.',
+ },
+ }]
+
+ def _real_extract(self, url):
+ display_id, article_id = re.match(self._VALID_URL, url).groups()
+ webpage = self._download_webpage(url, display_id)
+ mobj = re.search(r"player\.loadRevision3Item\('([^']+)'\s*,\s*(\d+)\);", webpage)
+ if mobj:
+ playlist_type, playlist_id = mobj.groups()
+ return self.url_result(
+ 'revision3:%s:%s' % (playlist_type, playlist_id), 'Revision3Embed', playlist_id)
+ else:
+ entries = [self.url_result('revision3:video_id:%s' % video_id, 'Revision3Embed', video_id) for video_id in re.findall(
+ r'<iframe[^>]+src=[\'"](?:https?:)?//api\.seekernetwork\.com/player/embed\?videoId=(\d+)', webpage)]
+ return self.playlist_result(
+ entries, article_id, self._og_search_title(webpage), self._og_search_description(webpage))
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .jwplatform import JWPlatformBaseIE
+from ..compat import compat_parse_qs
+from ..utils import (
+ ExtractorError,
+ parse_duration,
+)
+
+
+class SendtoNewsIE(JWPlatformBaseIE):
+ _VALID_URL = r'https?://embed\.sendtonews\.com/player/embed\.php\?(?P<query>[^#]+)'
+
+ _TEST = {
+ # From http://cleveland.cbslocal.com/2016/05/16/indians-score-season-high-15-runs-in-blowout-win-over-reds-rapid-reaction/
+ 'url': 'http://embed.sendtonews.com/player/embed.php?SK=GxfCe0Zo7D&MK=175909&PK=5588&autoplay=on&sound=yes',
+ 'info_dict': {
+ 'id': 'GxfCe0Zo7D-175909-5588',
+ 'ext': 'mp4',
+ 'title': 'Recap: CLE 15, CIN 6',
+ 'description': '5/16/16: Indians\' bats explode for 15 runs in a win',
+ 'duration': 49,
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
+ }
+
+ _URL_TEMPLATE = '//embed.sendtonews.com/player/embed.php?SK=%s&MK=%s&PK=%s'
+
+ @classmethod
+ def _extract_url(cls, webpage):
+ mobj = re.search(r'''(?x)<script[^>]+src=([\'"])
+ (?:https?:)?//embed\.sendtonews\.com/player/responsiveembed\.php\?
+ .*\bSC=(?P<SC>[0-9a-zA-Z-]+).*
+ \1>''', webpage)
+ if mobj:
+ sk, mk, pk = mobj.group('SC').split('-')
+ return cls._URL_TEMPLATE % (sk, mk, pk)
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ params = compat_parse_qs(mobj.group('query'))
+
+ if 'SK' not in params or 'MK' not in params or 'PK' not in params:
+ raise ExtractorError('Invalid URL', expected=True)
+
+ video_id = '-'.join([params['SK'][0], params['MK'][0], params['PK'][0]])
+
+ webpage = self._download_webpage(url, video_id)
+
+ jwplayer_data_str = self._search_regex(
+ r'jwplayer\("[^"]+"\)\.setup\((.+?)\);', webpage, 'JWPlayer data')
+ js_vars = {
+ 'w': 1024,
+ 'h': 768,
+ 'modeVar': 'html5',
+ }
+ for name, val in js_vars.items():
+ js_val = '%d' % val if isinstance(val, int) else '"%s"' % val
+ jwplayer_data_str = jwplayer_data_str.replace(':%s,' % name, ':%s,' % js_val)
+
+ info_dict = self._parse_jwplayer_data(
+ self._parse_json(jwplayer_data_str, video_id),
+ video_id, require_title=False, rtmp_params={'no_resume': True})
+
+ title = self._html_search_regex(
+ r'<div[^>]+class="embedTitle">([^<]+)</div>', webpage, 'title')
+ description = self._html_search_regex(
+ r'<div[^>]+class="embedSubTitle">([^<]+)</div>', webpage,
+ 'description', fatal=False)
+ duration = parse_duration(self._html_search_regex(
+ r'<div[^>]+class="embedDetails">([0-9:]+)', webpage,
+ 'duration', fatal=False))
+
+ info_dict.update({
+ 'title': title,
+ 'description': description,
+ 'duration': duration,
+ })
+
+ return info_dict
+++ /dev/null
-# coding: utf-8
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import (
- unified_strdate,
- parse_duration,
- int_or_none,
-)
-
-
-class SexyKarmaIE(InfoExtractor):
- IE_DESC = 'Sexy Karma and Watch Indian Porn'
- _VALID_URL = r'https?://(?:www\.)?(?:sexykarma\.com|watchindianporn\.net)/(?:[^/]+/)*video/(?P<display_id>[^/]+)-(?P<id>[a-zA-Z0-9]+)\.html'
- _TESTS = [{
- 'url': 'http://www.sexykarma.com/gonewild/video/taking-a-quick-pee-yHI70cOyIHt.html',
- 'md5': 'b9798e7d1ef1765116a8f516c8091dbd',
- 'info_dict': {
- 'id': 'yHI70cOyIHt',
- 'display_id': 'taking-a-quick-pee',
- 'ext': 'mp4',
- 'title': 'Taking a quick pee.',
- 'thumbnail': 're:^https?://.*\.jpg$',
- 'uploader': 'wildginger7',
- 'upload_date': '20141008',
- 'duration': 22,
- 'view_count': int,
- 'comment_count': int,
- 'categories': list,
- 'age_limit': 18,
- }
- }, {
- 'url': 'http://www.sexykarma.com/gonewild/video/pot-pixie-tribute-8Id6EZPbuHf.html',
- 'md5': 'dd216c68d29b49b12842b9babe762a5d',
- 'info_dict': {
- 'id': '8Id6EZPbuHf',
- 'display_id': 'pot-pixie-tribute',
- 'ext': 'mp4',
- 'title': 'pot_pixie tribute',
- 'thumbnail': 're:^https?://.*\.jpg$',
- 'uploader': 'banffite',
- 'upload_date': '20141013',
- 'duration': 16,
- 'view_count': int,
- 'comment_count': int,
- 'categories': list,
- 'age_limit': 18,
- }
- }, {
- 'url': 'http://www.watchindianporn.net/video/desi-dancer-namrata-stripping-completely-nude-and-dancing-on-a-hot-number-dW2mtctxJfs.html',
- 'md5': '9afb80675550406ed9a63ac2819ef69d',
- 'info_dict': {
- 'id': 'dW2mtctxJfs',
- 'display_id': 'desi-dancer-namrata-stripping-completely-nude-and-dancing-on-a-hot-number',
- 'ext': 'mp4',
- 'title': 'Desi dancer namrata stripping completely nude and dancing on a hot number',
- 'thumbnail': 're:^https?://.*\.jpg$',
- 'uploader': 'Don',
- 'upload_date': '20140213',
- 'duration': 83,
- 'view_count': int,
- 'comment_count': int,
- 'categories': list,
- 'age_limit': 18,
- }
- }]
-
- def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- video_id = mobj.group('id')
- display_id = mobj.group('display_id')
-
- webpage = self._download_webpage(url, display_id)
-
- video_url = self._html_search_regex(
- r"url: escape\('([^']+)'\)", webpage, 'url')
-
- title = self._html_search_regex(
- r'<h2 class="he2"><span>(.*?)</span>',
- webpage, 'title')
- thumbnail = self._html_search_regex(
- r'<span id="container"><img\s+src="([^"]+)"',
- webpage, 'thumbnail', fatal=False)
-
- uploader = self._html_search_regex(
- r'class="aupa">\s*(.*?)</a>',
- webpage, 'uploader')
- upload_date = unified_strdate(self._html_search_regex(
- r'Added: <strong>(.+?)</strong>', webpage, 'upload date', fatal=False))
-
- duration = parse_duration(self._search_regex(
- r'<td>Time:\s*</td>\s*<td align="right"><span>\s*(.+?)\s*</span>',
- webpage, 'duration', fatal=False))
-
- view_count = int_or_none(self._search_regex(
- r'<td>Views:\s*</td>\s*<td align="right"><span>\s*(\d+)\s*</span>',
- webpage, 'view count', fatal=False))
- comment_count = int_or_none(self._search_regex(
- r'<td>Comments:\s*</td>\s*<td align="right"><span>\s*(\d+)\s*</span>',
- webpage, 'comment count', fatal=False))
-
- categories = re.findall(
- r'<a href="[^"]+/search/video/desi"><span>([^<]+)</span></a>',
- webpage)
-
- return {
- 'id': video_id,
- 'display_id': display_id,
- 'url': video_url,
- 'title': title,
- 'thumbnail': thumbnail,
- 'uploader': uploader,
- 'upload_date': upload_date,
- 'duration': duration,
- 'view_count': view_count,
- 'comment_count': comment_count,
- 'categories': categories,
- 'age_limit': 18,
- }
import re
from .common import InfoExtractor
-from ..compat import compat_urllib_parse_urlencode
-from ..utils import sanitized_Request
+from ..utils import (
+ HEADRequest,
+ ExtractorError,
+ int_or_none,
+ update_url_query,
+ qualities,
+ get_element_by_attribute,
+ clean_html,
+)
class SinaIE(InfoExtractor):
- _VALID_URL = r'''(?x)https?://(.*?\.)?video\.sina\.com\.cn/
- (
- (.+?/(((?P<pseudo_id>\d+).html)|(.*?(\#|(vid=)|b/)(?P<id>\d+?)($|&|\-))))
- |
+ _VALID_URL = r'''(?x)https?://(?:.*?\.)?video\.sina\.com\.cn/
+ (?:
+ (?:view/|.*\#)(?P<video_id>\d+)|
+ .+?/(?P<pseudo_id>[^/?#]+)(?:\.s?html)|
# This is used by external sites like Weibo
- (api/sinawebApi/outplay.php/(?P<token>.+?)\.swf)
+ api/sinawebApi/outplay.php/(?P<token>.+?)\.swf
)
'''
_TESTS = [
{
- 'url': 'http://video.sina.com.cn/news/vlist/zt/chczlj2013/?opsubject_id=top12#110028898',
- 'md5': 'd65dd22ddcf44e38ce2bf58a10c3e71f',
+ 'url': 'http://video.sina.com.cn/news/spj/topvideoes20160504/?opsubject_id=top1#250576622',
+ 'md5': 'd38433e2fc886007729735650ae4b3e9',
'info_dict': {
- 'id': '110028898',
- 'ext': 'flv',
- 'title': '《中国新闻》 朝鲜要求巴拿马立即释放被扣船员',
+ 'id': '250576622',
+ 'ext': 'mp4',
+ 'title': '现场:克鲁兹宣布退选 特朗普将稳获提名',
}
},
{
'ext': 'flv',
'title': '军方提高对朝情报监视级别',
},
+ 'skip': 'the page does not exist or has been deleted',
+ },
+ {
+ 'url': 'http://video.sina.com.cn/view/250587748.html',
+ 'md5': '3d1807a25c775092aab3bc157fff49b4',
+ 'info_dict': {
+ 'id': '250587748',
+ 'ext': 'mp4',
+ 'title': '瞬间泪目：8年前汶川地震珍贵视频首曝光',
+ },
},
]
- def _extract_video(self, video_id):
- data = compat_urllib_parse_urlencode({'vid': video_id})
- url_doc = self._download_xml('http://v.iask.com/v_play.php?%s' % data,
- video_id, 'Downloading video url')
- image_page = self._download_webpage(
- 'http://interface.video.sina.com.cn/interface/common/getVideoImage.php?%s' % data,
- video_id, 'Downloading thumbnail info')
-
- return {'id': video_id,
- 'url': url_doc.find('./durl/url').text,
- 'ext': 'flv',
- 'title': url_doc.find('./vname').text,
- 'thumbnail': image_page.split('=')[1],
- }
-
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
- video_id = mobj.group('id')
- if mobj.group('token') is not None:
- # The video id is in the redirected url
- self.to_screen('Getting video id')
- request = sanitized_Request(url)
- request.get_method = lambda: 'HEAD'
- (_, urlh) = self._download_webpage_handle(request, 'NA', False)
- return self._real_extract(urlh.geturl())
- elif video_id is None:
- pseudo_id = mobj.group('pseudo_id')
- webpage = self._download_webpage(url, pseudo_id)
- video_id = self._search_regex(r'vid:\'(\d+?)\'', webpage, 'video id')
- return self._extract_video(video_id)
+ video_id = mobj.group('video_id')
+ if not video_id:
+ if mobj.group('token') is not None:
+ # The video id is in the redirected url
+ self.to_screen('Getting video id')
+ request = HEADRequest(url)
+ (_, urlh) = self._download_webpage_handle(request, 'NA', False)
+ return self._real_extract(urlh.geturl())
+ else:
+ pseudo_id = mobj.group('pseudo_id')
+ webpage = self._download_webpage(url, pseudo_id)
+ error = get_element_by_attribute('class', 'errtitle', webpage)
+ if error:
+ raise ExtractorError('%s said: %s' % (
+ self.IE_NAME, clean_html(error)), expected=True)
+ video_id = self._search_regex(
+ r"video_id\s*:\s*'(\d+)'", webpage, 'video id')
+
+ video_data = self._download_json(
+ 'http://s.video.sina.com.cn/video/h5play',
+ video_id, query={'video_id': video_id})
+ if video_data['code'] != 1:
+ raise ExtractorError('%s said: %s' % (
+ self.IE_NAME, video_data['message']), expected=True)
+ else:
+ video_data = video_data['data']
+ title = video_data['title']
+ description = video_data.get('description')
+ if description:
+ description = description.strip()
+
+ preference = qualities(['cif', 'sd', 'hd', 'fhd', 'ffd'])
+ formats = []
+ for quality_id, quality in video_data.get('videos', {}).get('mp4', {}).items():
+ file_api = quality.get('file_api')
+ file_id = quality.get('file_id')
+ if not file_api or not file_id:
+ continue
+ formats.append({
+ 'format_id': quality_id,
+ 'url': update_url_query(file_api, {'vid': file_id}),
+ 'preference': preference(quality_id),
+ 'ext': 'mp4',
+ })
+ self._sort_formats(formats)
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'description': description,
+ 'thumbnail': video_data.get('image'),
+ 'duration': int_or_none(video_data.get('length')),
+ 'timestamp': int_or_none(video_data.get('create_time')),
+ 'formats': formats,
+ }
+++ /dev/null
-from __future__ import unicode_literals
-
-import re
-
-from .common import InfoExtractor
-from ..utils import (
- ExtractorError,
- clean_html,
- determine_ext,
- int_or_none,
- js_to_json,
- parse_duration,
-)
-
-
-class SnagFilmsEmbedIE(InfoExtractor):
- _VALID_URL = r'https?://(?:(?:www|embed)\.)?snagfilms\.com/embed/player\?.*\bfilmId=(?P<id>[\da-f-]{36})'
- _TESTS = [{
- 'url': 'http://embed.snagfilms.com/embed/player?filmId=74849a00-85a9-11e1-9660-123139220831&w=500',
- 'md5': '2924e9215c6eff7a55ed35b72276bd93',
- 'info_dict': {
- 'id': '74849a00-85a9-11e1-9660-123139220831',
- 'ext': 'mp4',
- 'title': '#whilewewatch',
- }
- }, {
- # invalid labels, 360p is better that 480p
- 'url': 'http://www.snagfilms.com/embed/player?filmId=17ca0950-a74a-11e0-a92a-0026bb61d036',
- 'md5': '882fca19b9eb27ef865efeeaed376a48',
- 'info_dict': {
- 'id': '17ca0950-a74a-11e0-a92a-0026bb61d036',
- 'ext': 'mp4',
- 'title': 'Life in Limbo',
- }
- }, {
- 'url': 'http://www.snagfilms.com/embed/player?filmId=0000014c-de2f-d5d6-abcf-ffef58af0017',
- 'only_matching': True,
- }]
-
- @staticmethod
- def _extract_url(webpage):
- mobj = re.search(
- r'<iframe[^>]+?src=(["\'])(?P<url>(?:https?:)?//(?:embed\.)?snagfilms\.com/embed/player.+?)\1',
- webpage)
- if mobj:
- return mobj.group('url')
-
- def _real_extract(self, url):
- video_id = self._match_id(url)
-
- webpage = self._download_webpage(url, video_id)
-
- if '>This film is not playable in your area.<' in webpage:
- raise ExtractorError(
- 'Film %s is not playable in your area.' % video_id, expected=True)
-
- formats = []
- for source in self._parse_json(js_to_json(self._search_regex(
- r'(?s)sources:\s*(\[.+?\]),', webpage, 'json')), video_id):
- file_ = source.get('file')
- if not file_:
- continue
- type_ = source.get('type')
- ext = determine_ext(file_)
- format_id = source.get('label') or ext
- if all(v == 'm3u8' for v in (type_, ext)):
- formats.extend(self._extract_m3u8_formats(
- file_, video_id, 'mp4', m3u8_id='hls'))
- else:
- bitrate = int_or_none(self._search_regex(
- [r'(\d+)kbps', r'_\d{1,2}x\d{1,2}_(\d{3,})\.%s' % ext],
- file_, 'bitrate', default=None))
- height = int_or_none(self._search_regex(
- r'^(\d+)[pP]$', format_id, 'height', default=None))
- formats.append({
- 'url': file_,
- 'format_id': format_id,
- 'tbr': bitrate,
- 'height': height,
- })
- self._sort_formats(formats)
-
- title = self._search_regex(
- [r"title\s*:\s*'([^']+)'", r'<title>([^<]+)</title>'],
- webpage, 'title')
-
- return {
- 'id': video_id,
- 'title': title,
- 'formats': formats,
- }
-
-
-class SnagFilmsIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?snagfilms\.com/(?:films/title|show)/(?P<id>[^?#]+)'
- _TESTS = [{
- 'url': 'http://www.snagfilms.com/films/title/lost_for_life',
- 'md5': '19844f897b35af219773fd63bdec2942',
- 'info_dict': {
- 'id': '0000014c-de2f-d5d6-abcf-ffef58af0017',
- 'display_id': 'lost_for_life',
- 'ext': 'mp4',
- 'title': 'Lost for Life',
- 'description': 'md5:fbdacc8bb6b455e464aaf98bc02e1c82',
- 'thumbnail': 're:^https?://.*\.jpg',
- 'duration': 4489,
- 'categories': ['Documentary', 'Crime', 'Award Winning', 'Festivals']
- }
- }, {
- 'url': 'http://www.snagfilms.com/show/the_world_cut_project/india',
- 'md5': 'e6292e5b837642bbda82d7f8bf3fbdfd',
- 'info_dict': {
- 'id': '00000145-d75c-d96e-a9c7-ff5c67b20000',
- 'display_id': 'the_world_cut_project/india',
- 'ext': 'mp4',
- 'title': 'India',
- 'description': 'md5:5c168c5a8f4719c146aad2e0dfac6f5f',
- 'thumbnail': 're:^https?://.*\.jpg',
- 'duration': 979,
- 'categories': ['Documentary', 'Sports', 'Politics']
- }
- }, {
- # Film is not playable in your area.
- 'url': 'http://www.snagfilms.com/films/title/inside_mecca',
- 'only_matching': True,
- }, {
- # Film is not available.
- 'url': 'http://www.snagfilms.com/show/augie_alone/flirting',
- 'only_matching': True,
- }]
-
- def _real_extract(self, url):
- display_id = self._match_id(url)
-
- webpage = self._download_webpage(url, display_id)
-
- if ">Sorry, the Film you're looking for is not available.<" in webpage:
- raise ExtractorError(
- 'Film %s is not available.' % display_id, expected=True)
-
- film_id = self._search_regex(r'filmId=([\da-f-]{36})"', webpage, 'film id')
-
- snag = self._parse_json(
- self._search_regex(
- 'Snag\.page\.data\s*=\s*(\[.+?\]);', webpage, 'snag'),
- display_id)
-
- for item in snag:
- if item.get('data', {}).get('film', {}).get('id') == film_id:
- data = item['data']['film']
- title = data['title']
- description = clean_html(data.get('synopsis'))
- thumbnail = data.get('image')
- duration = int_or_none(data.get('duration') or data.get('runtime'))
- categories = [
- category['title'] for category in data.get('categories', [])
- if category.get('title')]
- break
- else:
- title = self._search_regex(
- r'itemprop="title">([^<]+)<', webpage, 'title')
- description = self._html_search_regex(
- r'(?s)<div itemprop="description" class="film-synopsis-inner ">(.+?)</div>',
- webpage, 'description', default=None) or self._og_search_description(webpage)
- thumbnail = self._og_search_thumbnail(webpage)
- duration = parse_duration(self._search_regex(
- r'<span itemprop="duration" class="film-duration strong">([^<]+)<',
- webpage, 'duration', fatal=False))
- categories = re.findall(r'<a href="/movies/[^"]+">([^<]+)</a>', webpage)
-
- return {
- '_type': 'url_transparent',
- 'url': 'http://embed.snagfilms.com/embed/player?filmId=%s' % film_id,
- 'id': film_id,
- 'display_id': display_id,
- 'title': title,
- 'description': description,
- 'thumbnail': thumbnail,
- 'duration': duration,
- 'categories': categories,
- }
formats = []
for height, video_url in zip(heights, video_urls):
path = compat_urllib_parse_urlparse(video_url).path
- _, quality = path.split('/')[4].split('_')[:2]
- f = {
+ m = re.search(r'/(?P<height>\d+)[pP]_(?P<tbr>\d+)[kK]', path)
+ if m:
+ tbr = int(m.group('tbr'))
+ height = int(m.group('height'))
+ else:
+ tbr = None
+ formats.append({
'url': video_url,
+ 'format_id': '%dp' % height,
'height': height,
- }
- tbr = self._search_regex(r'^(\d+)[Kk]$', quality, 'tbr', default=None)
- if tbr:
- f.update({
- 'tbr': int(tbr),
- 'format_id': '%dp' % height,
- })
- else:
- f['format_id'] = quality
- formats.append(f)
+ 'tbr': tbr,
+ })
self._sort_formats(formats)
age_limit = self._rta_search(webpage)
import re
from .common import InfoExtractor
-from ..utils import parse_filesize
+from ..utils import (
+ determine_ext,
+ js_to_json,
+ parse_iso8601,
+ parse_filesize,
+)
+
+
+class TagesschauPlayerIE(InfoExtractor):
+ IE_NAME = 'tagesschau:player'
+ _VALID_URL = r'https?://(?:www\.)?tagesschau\.de/multimedia/(?P<kind>audio|video)/(?P=kind)-(?P<id>\d+)~player(?:_[^/?#&]+)?\.html'
+
+ _TESTS = [{
+ 'url': 'http://www.tagesschau.de/multimedia/video/video-179517~player.html',
+ 'md5': '8d09548d5c15debad38bee3a4d15ca21',
+ 'info_dict': {
+ 'id': '179517',
+ 'ext': 'mp4',
+ 'title': 'Marie Kristin Boese, ARD Berlin, über den zukünftigen Kurs der AfD',
+ 'thumbnail': 're:^https?:.*\.jpg$',
+ 'formats': 'mincount:6',
+ },
+ }, {
+ 'url': 'https://www.tagesschau.de/multimedia/audio/audio-29417~player.html',
+ 'md5': '76e6eec6ebd40740671cf0a2c88617e5',
+ 'info_dict': {
+ 'id': '29417',
+ 'ext': 'mp3',
+ 'title': 'Trabi - Bye, bye Rennpappe',
+ 'thumbnail': 're:^https?:.*\.jpg$',
+ 'formats': 'mincount:2',
+ },
+ }, {
+ 'url': 'http://www.tagesschau.de/multimedia/audio/audio-29417~player_autoplay-true.html',
+ 'only_matching': True,
+ }]
+
+ _FORMATS = {
+ 'xs': {'quality': 0},
+ 's': {'width': 320, 'height': 180, 'quality': 1},
+ 'm': {'width': 512, 'height': 288, 'quality': 2},
+ 'l': {'width': 960, 'height': 540, 'quality': 3},
+ 'xl': {'width': 1280, 'height': 720, 'quality': 4},
+ 'xxl': {'quality': 5},
+ }
+
+ def _extract_via_api(self, kind, video_id):
+ info = self._download_json(
+ 'https://www.tagesschau.de/api/multimedia/{0}/{0}-{1}.json'.format(kind, video_id),
+ video_id)
+ title = info['headline']
+ formats = []
+ for media in info['mediadata']:
+ for format_id, format_url in media.items():
+ if determine_ext(format_url) == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ format_url, video_id, 'mp4',
+ entry_protocol='m3u8_native', m3u8_id='hls'))
+ else:
+ formats.append({
+ 'url': format_url,
+ 'format_id': format_id,
+ 'vcodec': 'none' if kind == 'audio' else None,
+ })
+ self._sort_formats(formats)
+ timestamp = parse_iso8601(info.get('date'))
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'timestamp': timestamp,
+ 'formats': formats,
+ }
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ video_id = mobj.group('id')
+
+ # kind = mobj.group('kind').lower()
+ # if kind == 'video':
+ # return self._extract_via_api(kind, video_id)
+
+ # JSON api does not provide some audio formats (e.g. ogg) thus
+ # extractiong audio via webpage
+
+ webpage = self._download_webpage(url, video_id)
+
+ title = self._og_search_title(webpage).strip()
+ formats = []
+
+ for media_json in re.findall(r'({src\s*:\s*["\']http[^}]+type\s*:[^}]+})', webpage):
+ media = self._parse_json(js_to_json(media_json), video_id, fatal=False)
+ if not media:
+ continue
+ src = media.get('src')
+ if not src:
+ return
+ quality = media.get('quality')
+ kind = media.get('type', '').split('/')[0]
+ ext = determine_ext(src)
+ f = {
+ 'url': src,
+ 'format_id': '%s_%s' % (quality, ext) if quality else ext,
+ 'ext': ext,
+ 'vcodec': 'none' if kind == 'audio' else None,
+ }
+ f.update(self._FORMATS.get(quality, {}))
+ formats.append(f)
+
+ self._sort_formats(formats)
+
+ thumbnail = self._og_search_thumbnail(webpage)
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'thumbnail': thumbnail,
+ 'formats': formats,
+ }
class TagesschauIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?tagesschau\.de/multimedia/(?:[^/]+/)*?[^/#?]+?(?P<id>-?[0-9]+)(?:~_[^/#?]+?)?\.html'
+ _VALID_URL = r'https?://(?:www\.)?tagesschau\.de/(?P<path>[^/]+/(?:[^/]+/)*?(?P<id>[^/#?]+?(?:-?[0-9]+)?))(?:~_?[^/#?]+?)?\.html'
_TESTS = [{
'url': 'http://www.tagesschau.de/multimedia/video/video-102143.html',
- 'md5': '917a228bc7df7850783bc47979673a09',
+ 'md5': 'f7c27a0eff3bfe8c7727e65f8fe1b1e6',
'info_dict': {
- 'id': '102143',
+ 'id': 'video-102143',
'ext': 'mp4',
'title': 'Regierungsumbildung in Athen: Neue Minister in Griechenland vereidigt',
- 'description': 'md5:171feccd9d9b3dd54d05d501568f6359',
+ 'description': '18.07.2015 20:10 Uhr',
'thumbnail': 're:^https?:.*\.jpg$',
},
}, {
'url': 'http://www.tagesschau.de/multimedia/sendung/ts-5727.html',
'md5': '3c54c1f6243d279b706bde660ceec633',
'info_dict': {
- 'id': '5727',
+ 'id': 'ts-5727',
'ext': 'mp4',
- 'description': 'md5:695c01bfd98b7e313c501386327aea59',
'title': 'Sendung: tagesschau \t04.12.2014 20:00 Uhr',
+ 'description': 'md5:695c01bfd98b7e313c501386327aea59',
+ 'thumbnail': 're:^https?:.*\.jpg$',
+ },
+ }, {
+ # exclusive audio
+ 'url': 'http://www.tagesschau.de/multimedia/audio/audio-29417.html',
+ 'md5': '76e6eec6ebd40740671cf0a2c88617e5',
+ 'info_dict': {
+ 'id': 'audio-29417',
+ 'ext': 'mp3',
+ 'title': 'Trabi - Bye, bye Rennpappe',
+ 'description': 'md5:8687dda862cbbe2cfb2df09b56341317',
'thumbnail': 're:^https?:.*\.jpg$',
},
}, {
- 'url': 'http://www.tagesschau.de/multimedia/politikimradio/audio-18407.html',
- 'md5': 'aef45de271c4bf0a5db834aa40bf774c',
+ # audio in article
+ 'url': 'http://www.tagesschau.de/inland/bnd-303.html',
+ 'md5': 'e0916c623e85fc1d2b26b78f299d3958',
'info_dict': {
- 'id': '18407',
+ 'id': 'bnd-303',
'ext': 'mp3',
- 'title': 'Flüchtlingsdebatte: Hitzig, aber wenig hilfreich',
- 'description': 'Flüchtlingsdebatte: Hitzig, aber wenig hilfreich',
+ 'title': 'Viele Baustellen für neuen BND-Chef',
+ 'description': 'md5:1e69a54be3e1255b2b07cdbce5bcd8b4',
'thumbnail': 're:^https?:.*\.jpg$',
},
+ }, {
+ 'url': 'http://www.tagesschau.de/inland/afd-parteitag-135.html',
+ 'info_dict': {
+ 'id': 'afd-parteitag-135',
+ 'title': 'Möchtegern-Underdog mit Machtanspruch',
+ },
+ 'playlist_count': 2,
}, {
'url': 'http://www.tagesschau.de/multimedia/sendung/tsg-3771.html',
'only_matching': True,
}, {
'url': 'http://www.tagesschau.de/multimedia/video/video-102303~_bab-sendung-211.html',
'only_matching': True,
+ }, {
+ 'url': 'http://www.tagesschau.de/100sekunden/index.html',
+ 'only_matching': True,
+ }, {
+ # playlist article with collapsing sections
+ 'url': 'http://www.tagesschau.de/wirtschaft/faq-freihandelszone-eu-usa-101.html',
+ 'only_matching': True,
}]
- _FORMATS = {
- 's': {'width': 256, 'height': 144, 'quality': 1},
- 'm': {'width': 512, 'height': 288, 'quality': 2},
- 'l': {'width': 960, 'height': 544, 'quality': 3},
- }
+ @classmethod
+ def suitable(cls, url):
+ return False if TagesschauPlayerIE.suitable(url) else super(TagesschauIE, cls).suitable(url)
+
+ def _extract_formats(self, download_text, media_kind):
+ links = re.finditer(
+ r'<div class="button" title="(?P<title>[^"]*)"><a href="(?P<url>[^"]+)">(?P<name>.+?)</a></div>',
+ download_text)
+ formats = []
+ for l in links:
+ link_url = l.group('url')
+ if not link_url:
+ continue
+ format_id = self._search_regex(
+ r'.*/[^/.]+\.([^/]+)\.[^/.]+$', link_url, 'format ID',
+ default=determine_ext(link_url))
+ format = {
+ 'format_id': format_id,
+ 'url': l.group('url'),
+ 'format_name': l.group('name'),
+ }
+ title = l.group('title')
+ if title:
+ if media_kind.lower() == 'video':
+ m = re.match(
+ r'''(?x)
+ Video:\s*(?P<vcodec>[a-zA-Z0-9/._-]+)\s*&\#10;
+ (?P<width>[0-9]+)x(?P<height>[0-9]+)px&\#10;
+ (?P<vbr>[0-9]+)kbps&\#10;
+ Audio:\s*(?P<abr>[0-9]+)kbps,\s*(?P<audio_desc>[A-Za-z\.0-9]+)&\#10;
+ Größe:\s*(?P<filesize_approx>[0-9.,]+\s+[a-zA-Z]*B)''',
+ title)
+ if m:
+ format.update({
+ 'format_note': m.group('audio_desc'),
+ 'vcodec': m.group('vcodec'),
+ 'width': int(m.group('width')),
+ 'height': int(m.group('height')),
+ 'abr': int(m.group('abr')),
+ 'vbr': int(m.group('vbr')),
+ 'filesize_approx': parse_filesize(m.group('filesize_approx')),
+ })
+ else:
+ m = re.match(
+ r'(?P<format>.+?)-Format\s*:\s*(?P<abr>\d+)kbps\s*,\s*(?P<note>.+)',
+ title)
+ if m:
+ format.update({
+ 'format_note': '%s, %s' % (m.group('format'), m.group('note')),
+ 'vcodec': 'none',
+ 'abr': int(m.group('abr')),
+ })
+ formats.append(format)
+ self._sort_formats(formats)
+ return formats
def _real_extract(self, url):
- video_id = self._match_id(url)
+ mobj = re.match(self._VALID_URL, url)
+ video_id = mobj.group('id') or mobj.group('path')
display_id = video_id.lstrip('-')
+
webpage = self._download_webpage(url, display_id)
- player_url = self._html_search_meta(
- 'twitter:player', webpage, 'player URL', default=None)
- if player_url:
- playerpage = self._download_webpage(
- player_url, display_id, 'Downloading player page')
-
- formats = []
- for media in re.finditer(
- r'''(?x)
- (?P<q_url>["\'])(?P<url>http://media.+?)(?P=q_url)
- ,\s*type:(?P<q_type>["\'])(?P<type>video|audio)/(?P<ext>.+?)(?P=q_type)
- (?:,\s*quality:(?P<q_quality>["\'])(?P<quality>.+?)(?P=q_quality))?
- ''', playerpage):
- url = media.group('url')
- type_ = media.group('type')
- ext = media.group('ext')
- res = media.group('quality')
- f = {
- 'format_id': '%s_%s' % (res, ext) if res else ext,
- 'url': url,
- 'ext': ext,
- 'vcodec': 'none' if type_ == 'audio' else None,
- }
- f.update(self._FORMATS.get(res, {}))
- formats.append(f)
- thumbnail = self._og_search_thumbnail(playerpage)
- title = self._og_search_title(webpage).strip()
- description = self._og_search_description(webpage).strip()
- else:
+ title = self._html_search_regex(
+ r'<span[^>]*class="headline"[^>]*>(.+?)</span>',
+ webpage, 'title', default=None) or self._og_search_title(webpage)
+
+ DOWNLOAD_REGEX = r'(?s)<p>Wir bieten dieses (?P<kind>Video|Audio) in folgenden Formaten zum Download an:</p>\s*<div class="controls">(?P<links>.*?)</div>\s*<p>'
+
+ webpage_type = self._og_search_property('type', webpage, default=None)
+ if webpage_type == 'website': # Article
+ entries = []
+ for num, (entry_title, media_kind, download_text) in enumerate(re.findall(
+ r'(?s)<p[^>]+class="infotext"[^>]*>\s*(?:<a[^>]+>)?\s*<strong>(.+?)</strong>.*?</p>.*?%s' % DOWNLOAD_REGEX,
+ webpage), 1):
+ entries.append({
+ 'id': '%s-%d' % (display_id, num),
+ 'title': '%s' % entry_title,
+ 'formats': self._extract_formats(download_text, media_kind),
+ })
+ if len(entries) > 1:
+ return self.playlist_result(entries, display_id, title)
+ formats = entries[0]['formats']
+ else: # Assume single video
download_text = self._search_regex(
- r'(?s)<p>Wir bieten dieses Video in folgenden Formaten zum Download an:</p>\s*<div class="controls">(.*?)</div>\s*<p>',
- webpage, 'download links')
- links = re.finditer(
- r'<div class="button" title="(?P<title>[^"]*)"><a href="(?P<url>[^"]+)">(?P<name>.+?)</a></div>',
- download_text)
- formats = []
- for l in links:
- format_id = self._search_regex(
- r'.*/[^/.]+\.([^/]+)\.[^/.]+', l.group('url'), 'format ID')
- format = {
- 'format_id': format_id,
- 'url': l.group('url'),
- 'format_name': l.group('name'),
- }
- m = re.match(
- r'''(?x)
- Video:\s*(?P<vcodec>[a-zA-Z0-9/._-]+)\s*&\#10;
- (?P<width>[0-9]+)x(?P<height>[0-9]+)px&\#10;
- (?P<vbr>[0-9]+)kbps&\#10;
- Audio:\s*(?P<abr>[0-9]+)kbps,\s*(?P<audio_desc>[A-Za-z\.0-9]+)&\#10;
- Größe:\s*(?P<filesize_approx>[0-9.,]+\s+[a-zA-Z]*B)''',
- l.group('title'))
- if m:
- format.update({
- 'format_note': m.group('audio_desc'),
- 'vcodec': m.group('vcodec'),
- 'width': int(m.group('width')),
- 'height': int(m.group('height')),
- 'abr': int(m.group('abr')),
- 'vbr': int(m.group('vbr')),
- 'filesize_approx': parse_filesize(m.group('filesize_approx')),
- })
- formats.append(format)
- thumbnail = self._og_search_thumbnail(webpage)
- description = self._html_search_regex(
- r'(?s)<p class="teasertext">(.*?)</p>',
- webpage, 'description', default=None)
- title = self._html_search_regex(
- r'<span class="headline".*?>(.*?)</span>', webpage, 'title')
+ DOWNLOAD_REGEX, webpage, 'download links', group='links')
+ media_kind = self._search_regex(
+ DOWNLOAD_REGEX, webpage, 'media kind', default='Video', group='kind')
+ formats = self._extract_formats(download_text, media_kind)
+ thumbnail = self._og_search_thumbnail(webpage)
+ description = self._html_search_regex(
+ r'(?s)<p class="teasertext">(.*?)</p>',
+ webpage, 'description', default=None)
self._sort_formats(formats)
_TEST = {
'url': 'https://www.teachingchannel.org/videos/teacher-teaming-evolution',
+ 'md5': '3d6361864d7cac20b57c8784da17166f',
'info_dict': {
'id': 'F3bnlzbToeI6pLEfRyrlfooIILUjz4nM',
'ext': 'mp4',
'duration': 422.255,
},
'params': {
- # m3u8 download
'skip_download': True,
},
+ 'add_ie': ['Ooyala'],
}
def _real_extract(self, url):
preload_codes = self._html_search_regex(
r'(function.+)setTimeout\(function\(\)\{playlist',
webpage, 'preload codes')
- base64_fragments = re.findall(r'"([a-zA-z0-9+/=]+)"', preload_codes)
+ base64_fragments = re.findall(r'"([a-zA-Z0-9+/=]+)"', preload_codes)
base64_fragments.remove('init')
def _check_sequence(cur_fragments):
'''
_TESTS = [{
'url': 'http://www.ted.com/talks/dan_dennett_on_our_consciousness.html',
- 'md5': 'fc94ac279feebbce69f21c0c6ee82810',
+ 'md5': '0de43ac406aa3e4ea74b66c9c7789b13',
'info_dict': {
'id': '102',
'ext': 'mp4',
'consciousness, but that half the time our brains are '
'actively fooling us.'),
'uploader': 'Dan Dennett',
- 'width': 854,
+ 'width': 853,
'duration': 1308,
}
}, {
'url': 'http://www.ted.com/watch/ted-institute/ted-bcg/vishal-sikka-the-beauty-and-power-of-algorithms',
- 'md5': '226f4fb9c62380d11b7995efa4c87994',
+ 'md5': 'b899ac15e345fb39534d913f7606082b',
'info_dict': {
- 'id': 'vishal-sikka-the-beauty-and-power-of-algorithms',
+ 'id': 'tSVI8ta_P4w',
'ext': 'mp4',
'title': 'Vishal Sikka: The beauty and power of algorithms',
'thumbnail': 're:^https?://.+\.jpg',
- 'description': 'Adaptive, intelligent, and consistent, algorithms are emerging as the ultimate app for everything from matching consumers to products to assessing medical diagnoses. Vishal Sikka shares his appreciation for the algorithm, charting both its inherent beauty and its growing power.',
- }
+ 'description': 'md5:6261fdfe3e02f4f579cbbfc00aff73f4',
+ 'upload_date': '20140122',
+ 'uploader_id': 'TEDInstitute',
+ 'uploader': 'TED Institute',
+ },
+ 'add_ie': ['Youtube'],
}, {
'url': 'http://www.ted.com/talks/gabby_giffords_and_mark_kelly_be_passionate_be_courageous_be_your_best',
+ 'md5': '71b3ab2f4233012dce09d515c9c39ce2',
'info_dict': {
'id': '1972',
'ext': 'mp4',
}]
_NATIVE_FORMATS = {
- 'low': {'preference': 1, 'width': 320, 'height': 180},
- 'medium': {'preference': 2, 'width': 512, 'height': 288},
- 'high': {'preference': 3, 'width': 854, 'height': 480},
+ 'low': {'width': 320, 'height': 180},
+ 'medium': {'width': 512, 'height': 288},
+ 'high': {'width': 854, 'height': 480},
}
def _extract_info(self, webpage):
if finfo:
f.update(finfo)
+ http_url = None
for format_id, resources in talk_info['resources'].items():
if format_id == 'h264':
for resource in resources:
+ h264_url = resource.get('file')
+ if not h264_url:
+ continue
bitrate = int_or_none(resource.get('bitrate'))
formats.append({
- 'url': resource['file'],
+ 'url': h264_url,
'format_id': '%s-%sk' % (format_id, bitrate),
'tbr': bitrate,
})
+ if re.search('\d+k', h264_url):
+ http_url = h264_url
elif format_id == 'rtmp':
streamer = talk_info.get('streamer')
if not streamer:
'tbr': int_or_none(resource.get('bitrate')),
})
elif format_id == 'hls':
- hls_formats = self._extract_m3u8_formats(
- resources.get('stream'), video_name, 'mp4', m3u8_id=format_id)
- for f in hls_formats:
- if f.get('format_id') == 'hls-meta':
- continue
- if not f.get('height'):
- f['vcodec'] = 'none'
- else:
- f['acodec'] = 'none'
- formats.extend(hls_formats)
+ formats.extend(self._extract_m3u8_formats(
+ resources.get('stream'), video_name, 'mp4', m3u8_id=format_id, fatal=False))
+
+ m3u8_formats = list(filter(
+ lambda f: f.get('protocol') == 'm3u8' and f.get('vcodec') != 'none' and f.get('resolution') != 'multiple',
+ formats))
+ if http_url:
+ for m3u8_format in m3u8_formats:
+ bitrate = self._search_regex(r'(\d+k)', m3u8_format['url'], 'bitrate', default=None)
+ if not bitrate:
+ continue
+ f = m3u8_format.copy()
+ f.update({
+ 'url': re.sub(r'\d+k', bitrate, http_url),
+ 'format_id': m3u8_format['format_id'].replace('hls', 'http'),
+ 'protocol': 'http',
+ })
+ formats.append(f)
audio_download = talk_info.get('audioDownload')
if audio_download:
'url': audio_download,
'format_id': 'audio',
'vcodec': 'none',
- 'preference': -0.5,
})
self._sort_formats(formats)
config_json = self._html_search_regex(
r'"pages\.jwplayer"\s*,\s*({.+?})\s*\)\s*</script>',
- webpage, 'config')
+ webpage, 'config', default=None)
+ if not config_json:
+ embed_url = self._search_regex(
+ r"<iframe[^>]+class='pages-video-embed__video__object'[^>]+src='([^']+)'", webpage, 'embed url')
+ return self.url_result(self._proto_relative_url(embed_url))
config = json.loads(config_json)['config']
video_url = config['video']['url']
thumbnail = config.get('image', {}).get('url')
from __future__ import unicode_literals
from .common import InfoExtractor
-from ..utils import remove_end
+from ..utils import (
+ determine_ext,
+ remove_end,
+)
class TelegraafIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?telegraaf\.nl/tv/(?:[^/]+/)+(?P<id>\d+)/[^/]+\.html'
_TEST = {
'url': 'http://www.telegraaf.nl/tv/nieuws/binnenland/24353229/__Tikibad_ontruimd_wegens_brand__.html',
- 'md5': '83245a9779bcc4a24454bfd53c65b6dc',
'info_dict': {
'id': '24353229',
'ext': 'mp4',
'thumbnail': 're:^https?://.*\.jpg$',
'duration': 33,
},
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
}
def _real_extract(self, url):
- playlist_id = self._match_id(url)
+ video_id = self._match_id(url)
- webpage = self._download_webpage(url, playlist_id)
+ webpage = self._download_webpage(url, video_id)
+ player_url = self._html_search_regex(
+ r'<iframe[^>]+src="([^"]+")', webpage, 'player URL')
+ player_page = self._download_webpage(
+ player_url, video_id, note='Download player webpage')
playlist_url = self._search_regex(
- r"iframe\.loadPlayer\('([^']+)'", webpage, 'player')
+ r'playlist\s*:\s*"([^"]+)"', player_page, 'playlist URL')
+ playlist_data = self._download_json(playlist_url, video_id)
+
+ item = playlist_data['items'][0]
+ formats = []
+ locations = item['locations']
+ for location in locations.get('adaptive', []):
+ manifest_url = location['src']
+ ext = determine_ext(manifest_url)
+ if ext == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ manifest_url, video_id, ext='mp4', m3u8_id='hls'))
+ elif ext == 'mpd':
+ # TODO: Current DASH formats are broken - $Time$ pattern in
+ # <SegmentTemplate> not implemented yet
+ continue
+ else:
+ self.report_warning('Unknown adaptive format %s' % ext)
+ for location in locations.get('progressive', []):
+ formats.append({
+ 'url': location['sources'][0]['src'],
+ 'width': location.get('width'),
+ 'height': location.get('height'),
+ 'format_id': 'http-%s' % location['label'],
+ })
+
+ self._sort_formats(formats)
- entries = self._extract_xspf_playlist(playlist_url, playlist_id)
title = remove_end(self._og_search_title(webpage), ' - VIDEO')
description = self._og_search_description(webpage)
+ duration = item.get('duration')
+ thumbnail = item.get('poster')
- return self.playlist_result(entries, playlist_id, title, description)
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'description': description,
+ 'formats': formats,
+ 'duration': duration,
+ 'thumbnail': thumbnail,
+ }
class TF1IE(InfoExtractor):
"""TF1 uses the wat.tv player."""
- _VALID_URL = r'https?://(?:(?:videos|www|lci)\.tf1|www\.tfou)\.fr/(?:[^/]+/)*(?P<id>.+?)\.html'
+ _VALID_URL = r'https?://(?:(?:videos|www|lci)\.tf1|(?:www\.)?(?:tfou|ushuaiatv|histoire|tvbreizh))\.fr/(?:[^/]+/)*(?P<id>[^/?#.]+)'
_TESTS = [{
'url': 'http://videos.tf1.fr/auto-moto/citroen-grand-c4-picasso-2013-presentation-officielle-8062060.html',
'info_dict': {
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
wat_id = self._html_search_regex(
- r'(["\'])(?:https?:)?//www\.wat\.tv/embedframe/.*?(?P<id>\d{8})(?:#.*?)?\1',
+ r'(["\'])(?:https?:)?//www\.wat\.tv/embedframe/.*?(?P<id>\d{8}).*?\1',
webpage, 'wat id', group='id')
return self.url_result('wat:%s' % wat_id, 'Wat')
'only_matching': True,
}]
+ @classmethod
+ def _extract_urls(cls, webpage):
+ m = re.search(
+ r'''(?x)
+ <meta\s+
+ property=(["'])(?:og:video(?::(?:secure_)?url)?|twitter:player)\1\s+
+ content=(["'])(?P<url>https?://player\.theplatform\.com/p/.+?)\2
+ ''', webpage)
+ if m:
+ return [m.group('url')]
+
+ matches = re.findall(
+ r'<(?:iframe|script)[^>]+src=(["\'])((?:https?:)?//player\.theplatform\.com/p/.+?)\1', webpage)
+ if matches:
+ return list(zip(*matches))[1]
+
@staticmethod
def _sign_url(url, sig_key, sig_secret, life=600, include_qs=False):
flags = '10' if include_qs else '00'
def str_to_hex(str):
return binascii.b2a_hex(str.encode('ascii')).decode('ascii')
- def hex_to_str(hex):
- return binascii.a2b_hex(hex)
+ def hex_to_bytes(hex):
+ return binascii.a2b_hex(hex.encode('ascii'))
relative_path = re.match(r'https?://link.theplatform.com/s/([^?]+)', url).group(1)
- clear_text = hex_to_str(flags + expiration_date + str_to_hex(relative_path))
+ clear_text = hex_to_bytes(flags + expiration_date + str_to_hex(relative_path))
checksum = hmac.new(sig_key.encode('ascii'), clear_text, hashlib.sha1).hexdigest()
sig = flags + expiration_date + checksum + str_to_hex(sig_secret)
return '%s&sig=%s' % (url, sig)
'timestamp': 1391824260,
'duration': 467.0,
'categories': ['MSNBC/Issues/Democrats', 'MSNBC/Issues/Elections/Election 2016'],
+ 'uploader': 'NBCU-NEWS',
},
}
--- /dev/null
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+ determine_ext,
+ js_to_json,
+ mimetype2ext,
+)
+
+
+class ThreeQSDNIE(InfoExtractor):
+ IE_NAME = '3qsdn'
+ IE_DESC = '3Q SDN'
+ _VALID_URL = r'https?://playout\.3qsdn\.com/(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
+ _TESTS = [{
+ # ondemand from http://www.philharmonie.tv/veranstaltung/26/
+ 'url': 'http://playout.3qsdn.com/0280d6b9-1215-11e6-b427-0cc47a188158?protocol=http',
+ 'md5': 'ab040e37bcfa2e0c079f92cb1dd7f6cd',
+ 'info_dict': {
+ 'id': '0280d6b9-1215-11e6-b427-0cc47a188158',
+ 'ext': 'mp4',
+ 'title': '0280d6b9-1215-11e6-b427-0cc47a188158',
+ 'is_live': False,
+ },
+ 'expected_warnings': ['Failed to download MPD manifest'],
+ }, {
+ # live video stream
+ 'url': 'https://playout.3qsdn.com/d755d94b-4ab9-11e3-9162-0025907ad44f?js=true',
+ 'info_dict': {
+ 'id': 'd755d94b-4ab9-11e3-9162-0025907ad44f',
+ 'ext': 'mp4',
+ 'title': 'd755d94b-4ab9-11e3-9162-0025907ad44f',
+ 'is_live': False,
+ },
+ }, {
+ # live audio stream
+ 'url': 'http://playout.3qsdn.com/9edf36e0-6bf2-11e2-a16a-9acf09e2db48',
+ 'only_matching': True,
+ }, {
+ # live audio stream with some 404 URLs
+ 'url': 'http://playout.3qsdn.com/ac5c3186-777a-11e2-9c30-9acf09e2db48',
+ 'only_matching': True,
+ }, {
+ # geo restricted with 'This content is not available in your country'
+ 'url': 'http://playout.3qsdn.com/d63a3ffe-75e8-11e2-9c30-9acf09e2db48',
+ 'only_matching': True,
+ }, {
+ # geo restricted with 'playout.3qsdn.com/forbidden'
+ 'url': 'http://playout.3qsdn.com/8e330f26-6ae2-11e2-a16a-9acf09e2db48',
+ 'only_matching': True,
+ }, {
+ # live video with rtmp link
+ 'url': 'https://playout.3qsdn.com/6092bb9e-8f72-11e4-a173-002590c750be',
+ 'only_matching': True,
+ }]
+
+ @staticmethod
+ def _extract_url(webpage):
+ mobj = re.search(
+ r'<iframe[^>]+\b(?:data-)?src=(["\'])(?P<url>%s.*?)\1' % ThreeQSDNIE._VALID_URL, webpage)
+ if mobj:
+ return mobj.group('url')
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+
+ js = self._download_webpage(
+ 'http://playout.3qsdn.com/%s' % video_id, video_id,
+ query={'js': 'true'})
+
+ if any(p in js for p in (
+ '>This content is not available in your country',
+ 'playout.3qsdn.com/forbidden')):
+ self.raise_geo_restricted()
+
+ stream_content = self._search_regex(
+ r'streamContent\s*:\s*(["\'])(?P<content>.+?)\1', js,
+ 'stream content', default='demand', group='content')
+
+ live = stream_content == 'live'
+
+ stream_type = self._search_regex(
+ r'streamType\s*:\s*(["\'])(?P<type>audio|video)\1', js,
+ 'stream type', default='video', group='type')
+
+ formats = []
+ urls = set()
+
+ def extract_formats(item_url, item={}):
+ if not item_url or item_url in urls:
+ return
+ urls.add(item_url)
+ type_ = item.get('type')
+ ext = determine_ext(item_url, default_ext=None)
+ if type_ == 'application/dash+xml' or ext == 'mpd':
+ formats.extend(self._extract_mpd_formats(
+ item_url, video_id, mpd_id='mpd', fatal=False))
+ elif type_ in ('application/vnd.apple.mpegURL', 'application/x-mpegurl') or ext == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ item_url, video_id, 'mp4',
+ entry_protocol='m3u8' if live else 'm3u8_native',
+ m3u8_id='hls', fatal=False))
+ elif ext == 'f4m':
+ formats.extend(self._extract_f4m_formats(
+ item_url, video_id, f4m_id='hds', fatal=False))
+ else:
+ if not self._is_valid_url(item_url, video_id):
+ return
+ formats.append({
+ 'url': item_url,
+ 'format_id': item.get('quality'),
+ 'ext': 'mp4' if item_url.startswith('rtsp') else mimetype2ext(type_) or ext,
+ 'vcodec': 'none' if stream_type == 'audio' else None,
+ })
+
+ for item_js in re.findall(r'({.*?\b(?:src|source)\s*:\s*["\'].+?})', js):
+ f = self._parse_json(
+ item_js, video_id, transform_source=js_to_json, fatal=False)
+ if not f:
+ continue
+ extract_formats(f.get('src'), f)
+
+ # More relaxed version to collect additional URLs and acting
+ # as a future-proof fallback
+ for _, src in re.findall(r'\b(?:src|source)\s*:\s*(["\'])((?:https?|rtsp)://.+?)\1', js):
+ extract_formats(src)
+
+ self._sort_formats(formats)
+
+ title = self._live_title(video_id) if live else video_id
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'is_live': live,
+ 'formats': formats,
+ }
-# -*- coding: utf-8 -*-
+# coding: utf-8
from __future__ import unicode_literals
import re
from .common import InfoExtractor
-class TvpIE(InfoExtractor):
- IE_NAME = 'tvp.pl'
- _VALID_URL = r'https?://(?:vod|www)\.tvp\.pl/.*/(?P<id>\d+)$'
+class TVPIE(InfoExtractor):
+ IE_NAME = 'tvp'
+ IE_DESC = 'Telewizja Polska'
+ _VALID_URL = r'https?://[^/]+\.tvp\.(?:pl|info)/(?:(?!\d+/)[^/]+/)*(?P<id>\d+)'
_TESTS = [{
- 'url': 'http://vod.tvp.pl/filmy-fabularne/filmy-za-darmo/ogniem-i-mieczem/wideo/odc-2/4278035',
- 'md5': 'cdd98303338b8a7f7abab5cd14092bf2',
- 'info_dict': {
- 'id': '4278035',
- 'ext': 'wmv',
- 'title': 'Ogniem i mieczem, odc. 2',
- },
- }, {
- 'url': 'http://vod.tvp.pl/seriale/obyczajowe/czas-honoru/sezon-1-1-13/i-seria-odc-13/194536',
+ 'url': 'http://vod.tvp.pl/194536/i-seria-odc-13',
'md5': '8aa518c15e5cc32dfe8db400dc921fbb',
'info_dict': {
'id': '194536',
},
}, {
'url': 'http://vod.tvp.pl/seriale/obyczajowe/na-sygnale/sezon-2-27-/odc-39/17834272',
- 'md5': 'c3b15ed1af288131115ff17a17c19dda',
- 'info_dict': {
- 'id': '17834272',
- 'ext': 'mp4',
- 'title': 'Na sygnale, odc. 39',
- },
+ 'only_matching': True,
+ }, {
+ 'url': 'http://wiadomosci.tvp.pl/25169746/24052016-1200',
+ 'only_matching': True,
+ }, {
+ 'url': 'http://krakow.tvp.pl/25511623/25lecie-mck-wyjatkowe-miejsce-na-mapie-krakowa',
+ 'only_matching': True,
+ }, {
+ 'url': 'http://teleexpress.tvp.pl/25522307/wierni-wzieli-udzial-w-procesjach',
+ 'only_matching': True,
+ }, {
+ 'url': 'http://sport.tvp.pl/25522165/krychowiak-uspokaja-w-sprawie-kontuzji-dwa-tygodnie-to-maksimum',
+ 'only_matching': True,
+ }, {
+ 'url': 'http://www.tvp.info/25511919/trwa-rewolucja-wladza-zdecydowala-sie-na-pogwalcenie-konstytucji',
+ 'only_matching': True,
}]
def _real_extract(self, url):
}
-class TvpSeriesIE(InfoExtractor):
- IE_NAME = 'tvp.pl:Series'
+class TVPSeriesIE(InfoExtractor):
+ IE_NAME = 'tvp:series'
_VALID_URL = r'https?://vod\.tvp\.pl/(?:[^/]+/){2}(?P<id>[^/]+)/?$'
_TESTS = [{
videos_paths = re.findall(
'(?s)class="shortTitle">.*?href="(/[^"]+)', playlist)
entries = [
- self.url_result('http://vod.tvp.pl%s' % v_path, ie=TvpIE.ie_key())
+ self.url_result('http://vod.tvp.pl%s' % v_path, ie=TVPIE.ie_key())
for v_path in videos_paths]
return {
title = self._og_search_title(webpage)
description = self._html_search_regex(
- r'<span itemprop="description">([^<]+)</span>', webpage, 'description', fatal=False)
+ r'<(p|span)[^>]+itemprop="description"[^>]*>(?P<description>[^<]+)</\1>',
+ webpage, 'description', fatal=False, group='description')
thumbnail = self._og_search_thumbnail(webpage)
duration = int_or_none(self._og_search_property(
'duration', webpage, 'duration', fatal=False))
'title': '«Wir müssen mutig nach vorne schauen»',
'description': 'Kein Land sei innovativer als die Schweiz, sagte Johann Schneider-Ammann in seiner Neujahrsansprache. Das Land müsse aber seine Hausaufgaben machen.',
'thumbnail': 'http://www.20min.ch/images/content/2/2/0/22050469/10/teaserbreit.jpg'
- }
+ },
+ 'skip': '"This video is no longer available" is shown both on the web page and in the downloaded file.',
+ }, {
+ # YouTube embed
+ 'url': 'http://www.20min.ch/ro/sports/football/story/Il-marque-une-bicyclette-de-plus-de-30-metres--21115184',
+ 'md5': 'cec64d59aa01c0ed9dbba9cf639dd82f',
+ 'info_dict': {
+ 'id': 'ivM7A7SpDOs',
+ 'ext': 'mp4',
+ 'title': 'GOLAZO DE CHILENA DE JAVI GÓMEZ, FINALISTA AL BALÓN DE CLM 2016',
+ 'description': 'md5:903c92fbf2b2f66c09de514bc25e9f5a',
+ 'upload_date': '20160424',
+ 'uploader': 'RTVCM Castilla-La Mancha',
+ 'uploader_id': 'RTVCM',
+ },
+ 'add_ie': ['Youtube'],
}, {
'url': 'http://www.20min.ch/videotv/?cid=44&vid=468738',
'only_matching': True,
webpage = self._download_webpage(url, display_id)
+ youtube_url = self._html_search_regex(
+ r'<iframe[^>]+src="((?:https?:)?//www\.youtube\.com/embed/[^"]+)"',
+ webpage, 'YouTube embed URL', default=None)
+ if youtube_url is not None:
+ return self.url_result(youtube_url, 'Youtube')
+
title = self._html_search_regex(
r'<h1>.*?<span>(.+?)</span></h1>',
webpage, 'title', default=None)
'title': 'Worlds Semifinals - Star Horn Royal Club vs. OMG',
},
'playlist_mincount': 12,
+ 'skip': 'HTTP Error 404: Not Found',
}
'title': 'ACRL Off Season - Sports Cars @ Nordschleife',
},
'playlist_mincount': 3,
+ 'skip': 'HTTP Error 404: Not Found',
}, {
'url': 'http://www.twitch.tv/tsm_theoddone/c/2349361',
'only_matching': True,
'nauth': access_token['token'],
'nauthsig': access_token['sig'],
})),
- item_id, 'mp4')
+ item_id, 'mp4', entry_protocol='m3u8_native')
self._prefer_source(formats)
info['formats'] = formats
}
-class TwitchBookmarksIE(TwitchPlaylistBaseIE):
- IE_NAME = 'twitch:bookmarks'
- _VALID_URL = r'%s/(?P<id>[^/]+)/profile/bookmarks/?(?:\#.*)?$' % TwitchBaseIE._VALID_URL_BASE
- _PLAYLIST_URL = '%s/api/bookmark/?user=%%s&offset=%%d&limit=%%d' % TwitchBaseIE._API_BASE
- _PLAYLIST_TYPE = 'bookmarks'
-
- _TEST = {
- 'url': 'http://www.twitch.tv/ognos/profile/bookmarks',
- 'info_dict': {
- 'id': 'ognos',
- 'title': 'Ognos',
- },
- 'playlist_mincount': 3,
- }
-
- def _extract_playlist_page(self, response):
- entries = []
- for bookmark in response.get('bookmarks', []):
- video = bookmark.get('video')
- if not video:
- continue
- entries.append(video['url'])
- return entries
-
-
class TwitchStreamIE(TwitchBaseIE):
IE_NAME = 'twitch:stream'
_VALID_URL = r'%s/(?P<id>[^/#?]+)/?(?:\#.*)?$' % TwitchBaseIE._VALID_URL_BASE
from .common import InfoExtractor
from ..compat import (
compat_HTTPError,
- compat_urllib_parse_urlencode,
compat_urllib_request,
compat_urlparse,
)
if enroll_url:
webpage = self._download_webpage(
combine_url(base_url, enroll_url),
- course_id, 'Enrolling in the course')
+ course_id, 'Enrolling in the course',
+ headers={'Referer': base_url})
if '>You have enrolled in' in webpage:
self.to_screen('%s: Successfully enrolled in the course' % course_id)
def _download_lecture(self, course_id, lecture_id):
return self._download_json(
- 'https://www.udemy.com/api-2.0/users/me/subscribed-courses/%s/lectures/%s?%s' % (
- course_id, lecture_id, compat_urllib_parse_urlencode({
- 'fields[lecture]': 'title,description,view_html,asset',
- 'fields[asset]': 'asset_type,stream_url,thumbnail_url,download_urls,data',
- })),
- lecture_id, 'Downloading lecture JSON')
+ 'https://www.udemy.com/api-2.0/users/me/subscribed-courses/%s/lectures/%s?'
+ % (course_id, lecture_id),
+ lecture_id, 'Downloading lecture JSON', query={
+ 'fields[lecture]': 'title,description,view_html,asset',
+ 'fields[asset]': 'asset_type,stream_url,thumbnail_url,download_urls,data',
+ })
def _handle_error(self, response):
if not isinstance(response, dict):
self._LOGIN_URL, None, 'Downloading login popup')
def is_logged(webpage):
- return any(p in webpage for p in ['href="https://www.udemy.com/user/logout/', '>Logout<'])
+ return any(re.search(p, webpage) for p in (
+ r'href=["\'](?:https://www\.udemy\.com)?/user/logout/',
+ r'>Logout<'))
# already logged in
if is_logged(login_popup):
'password': password,
})
- request = sanitized_Request(
- self._LOGIN_URL, urlencode_postdata(login_form))
- request.add_header('Referer', self._ORIGIN_URL)
- request.add_header('Origin', self._ORIGIN_URL)
-
response = self._download_webpage(
- request, None, 'Logging in as %s' % username)
+ self._LOGIN_URL, None, 'Logging in as %s' % username,
+ data=urlencode_postdata(login_form),
+ headers={
+ 'Referer': self._ORIGIN_URL,
+ 'Origin': self._ORIGIN_URL,
+ })
if not is_logged(response):
error = self._html_search_regex(
from __future__ import unicode_literals
import json
+import re
+
from .common import InfoExtractor
from ..utils import (
+ determine_ext,
+ int_or_none,
js_to_json,
- ExtractorError,
)
from ..compat import compat_urlparse
_VALID_URL = r'https?:' + _PROTOCOL_RELATIVE_VALID_URL
_TESTS = [{
'url': 'http://video.udn.com/embed/news/300040',
- 'md5': 'de06b4c90b042c128395a88f0384817e',
'info_dict': {
'id': '300040',
'ext': 'mp4',
'title': '生物老師男變女 全校挺"做自己"',
'thumbnail': 're:^https?://.*\.jpg$',
- }
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
}, {
'url': 'https://video.udn.com/embed/news/300040',
'only_matching': True,
page = self._download_webpage(url, video_id)
options = json.loads(js_to_json(self._html_search_regex(
- r'var options\s*=\s*([^;]+);', page, 'video urls dictionary')))
+ r'var\s+options\s*=\s*([^;]+);', page, 'video urls dictionary')))
video_urls = options['video']
if video_urls.get('youtube'):
return self.url_result(video_urls.get('youtube'), 'Youtube')
- try:
- del video_urls['youtube']
- except KeyError:
- pass
+ formats = []
+ for video_type, api_url in video_urls.items():
+ if not api_url:
+ continue
- formats = [{
- 'url': self._download_webpage(
+ video_url = self._download_webpage(
compat_urlparse.urljoin(url, api_url), video_id,
- 'retrieve url for %s video' % video_type),
- 'format_id': video_type,
- 'preference': 0 if video_type == 'mp4' else -1,
- } for video_type, api_url in video_urls.items() if api_url]
+ note='retrieve url for %s video' % video_type)
- if not formats:
- raise ExtractorError('No videos found', expected=True)
+ ext = determine_ext(video_url)
+ if ext == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ video_url, video_id, ext='mp4', m3u8_id='hls'))
+ elif ext == 'f4m':
+ formats.extend(self._extract_f4m_formats(
+ video_url, video_id, f4m_id='hds'))
+ else:
+ mobj = re.search(r'_(?P<height>\d+)p_(?P<tbr>\d+).mp4', video_url)
+ a_format = {
+ 'url': video_url,
+ # video_type may be 'mp4', which confuses YoutubeDL
+ 'format_id': 'http-' + video_type,
+ }
+ if mobj:
+ a_format.update({
+ 'height': int_or_none(mobj.group('height')),
+ 'tbr': int_or_none(mobj.group('tbr')),
+ })
+ formats.append(a_format)
self._sort_formats(formats)
- thumbnail = None
-
- if options.get('gallery') and len(options['gallery']):
- thumbnail = options['gallery'][0].get('original')
+ thumbnails = [{
+ 'url': img_url,
+ 'id': img_type,
+ } for img_type, img_url in options.get('gallery', [{}])[0].items() if img_url]
return {
'id': video_id,
'formats': formats,
'title': options['title'],
- 'thumbnail': thumbnail
+ 'thumbnails': thumbnails,
}
'format_id': format_id,
'quality': quality(format_id)
})
+ self._sort_formats(formats)
title = self._html_search_regex(
r'<title>UTV - (.*?)</', webpage, 'title')
from ..utils import (
int_or_none,
unified_strdate,
+ unescapeHTML,
)
class UstudioIE(InfoExtractor):
+ IE_NAME = 'ustudio'
_VALID_URL = r'https?://(?:(?:www|v1)\.)?ustudio\.com/video/(?P<id>[^/]+)/(?P<display_id>[^/?#&]+)'
_TEST = {
'url': 'http://ustudio.com/video/Uxu2my9bgSph/san_francisco_golden_gate_bridge',
}
def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- video_id = mobj.group('id')
- display_id = mobj.group('display_id')
+ video_id, display_id = re.match(self._VALID_URL, url).groups()
config = self._download_xml(
'http://v1.ustudio.com/embed/%s/ustudio/config.xml' % video_id,
def extract(kind):
return [{
- 'url': item.attrib['url'],
+ 'url': unescapeHTML(item.attrib['url']),
'width': int_or_none(item.get('width')),
'height': int_or_none(item.get('height')),
} for item in config.findall('./qualities/quality/%s' % kind) if item.get('url')]
'uploader': uploader,
'formats': formats,
}
+
+
+class UstudioEmbedIE(InfoExtractor):
+ IE_NAME = 'ustudio:embed'
+ _VALID_URL = r'https?://(?:(?:app|embed)\.)?ustudio\.com/embed/(?P<uid>[^/]+)/(?P<id>[^/]+)'
+ _TEST = {
+ 'url': 'http://app.ustudio.com/embed/DeN7VdYRDKhP/Uw7G1kMCe65T',
+ 'md5': '47c0be52a09b23a7f40de9469cec58f4',
+ 'info_dict': {
+ 'id': 'Uw7G1kMCe65T',
+ 'ext': 'mp4',
+ 'title': '5 Things IT Should Know About Video',
+ 'description': 'md5:93d32650884b500115e158c5677d25ad',
+ 'uploader_id': 'DeN7VdYRDKhP',
+ }
+ }
+
+ def _real_extract(self, url):
+ uploader_id, video_id = re.match(self._VALID_URL, url).groups()
+ video_data = self._download_json(
+ 'http://app.ustudio.com/embed/%s/%s/config.json' % (uploader_id, video_id),
+ video_id)['videos'][0]
+ title = video_data['name']
+
+ formats = []
+ for ext, qualities in video_data.get('transcodes', {}).items():
+ for quality in qualities:
+ quality_url = quality.get('url')
+ if not quality_url:
+ continue
+ height = int_or_none(quality.get('height'))
+ formats.append({
+ 'format_id': '%s-%dp' % (ext, height) if height else ext,
+ 'url': quality_url,
+ 'width': int_or_none(quality.get('width')),
+ 'height': height,
+ })
+ self._sort_formats(formats)
+
+ thumbnails = []
+ for image in video_data.get('images', []):
+ image_url = image.get('url')
+ if not image_url:
+ continue
+ thumbnails.append({
+ 'url': image_url,
+ })
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'description': video_data.get('description'),
+ 'duration': int_or_none(video_data.get('duration')),
+ 'uploader_id': uploader_id,
+ 'tags': video_data.get('keywords'),
+ 'thumbnails': thumbnails,
+ 'formats': formats,
+ }
'uploader': 'afp-news',
'duration': 123,
},
+ 'skip': 'This video has been deleted.',
},
{
'url': 'http://www.veoh.com/watch/v69525809F6Nc4frX',
import re
from .common import InfoExtractor
-from ..compat import compat_etree_fromstring
+from ..compat import (
+ compat_etree_fromstring,
+ compat_str,
+ compat_urlparse,
+)
from ..utils import (
ExtractorError,
int_or_none,
)
-class VevoIE(InfoExtractor):
+class VevoBaseIE(InfoExtractor):
+ def _extract_json(self, webpage, video_id, item):
+ return self._parse_json(
+ self._search_regex(
+ r'window\.__INITIAL_STORE__\s*=\s*({.+?});\s*</script>',
+ webpage, 'initial store'),
+ video_id)['default'][item]
+
+
+class VevoIE(VevoBaseIE):
'''
Accepts urls from vevo.com or in the format 'vevo:{id}'
(currently used by MTVIE and MySpaceIE)
'''
_VALID_URL = r'''(?x)
- (?:https?://www\.vevo\.com/watch/(?:[^/]+/(?:[^/]+/)?)?|
+ (?:https?://www\.vevo\.com/watch/(?!playlist|genre)(?:[^/]+/(?:[^/]+/)?)?|
https?://cache\.vevo\.com/m/html/embed\.html\?video=|
https?://videoplayer\.vevo\.com/embed/embedded\?videoId=|
vevo:)
'info_dict': {
'id': 'GB1101300280',
'ext': 'mp4',
- 'title': 'Somebody to Die For',
+ 'title': 'Hurts - Somebody to Die For',
+ 'timestamp': 1372057200,
'upload_date': '20130624',
'uploader': 'Hurts',
- 'timestamp': 1372057200,
+ 'track': 'Somebody to Die For',
+ 'artist': 'Hurts',
+ 'genre': 'Pop',
},
+ 'expected_warnings': ['Unable to download SMIL file'],
}, {
'note': 'v3 SMIL format',
'url': 'http://www.vevo.com/watch/cassadee-pope/i-wish-i-could-break-your-heart/USUV71302923',
'info_dict': {
'id': 'USUV71302923',
'ext': 'mp4',
- 'title': 'I Wish I Could Break Your Heart',
+ 'title': 'Cassadee Pope - I Wish I Could Break Your Heart',
+ 'timestamp': 1392796919,
'upload_date': '20140219',
'uploader': 'Cassadee Pope',
- 'timestamp': 1392796919,
+ 'track': 'I Wish I Could Break Your Heart',
+ 'artist': 'Cassadee Pope',
+ 'genre': 'Country',
},
+ 'expected_warnings': ['Unable to download SMIL file'],
}, {
'note': 'Age-limited video',
'url': 'https://www.vevo.com/watch/justin-timberlake/tunnel-vision-explicit/USRV81300282',
'info_dict': {
'id': 'USRV81300282',
'ext': 'mp4',
- 'title': 'Tunnel Vision (Explicit)',
- 'upload_date': '20130703',
+ 'title': 'Justin Timberlake - Tunnel Vision (Explicit)',
'age_limit': 18,
- 'uploader': 'Justin Timberlake',
'timestamp': 1372888800,
+ 'upload_date': '20130703',
+ 'uploader': 'Justin Timberlake',
+ 'track': 'Tunnel Vision (Explicit)',
+ 'artist': 'Justin Timberlake',
+ 'genre': 'Pop',
},
+ 'expected_warnings': ['Unable to download SMIL file'],
}, {
'note': 'No video_info',
'url': 'http://www.vevo.com/watch/k-camp-1/Till-I-Die/USUV71503000',
'info_dict': {
'id': 'USUV71503000',
'ext': 'mp4',
- 'title': 'Till I Die',
- 'upload_date': '20151207',
+ 'title': 'K Camp - Till I Die',
'age_limit': 18,
- 'uploader': 'K Camp',
'timestamp': 1449468000,
+ 'upload_date': '20151207',
+ 'uploader': 'K Camp',
+ 'track': 'Till I Die',
+ 'artist': 'K Camp',
+ 'genre': 'Rap/Hip-Hop',
+ },
+ }, {
+ 'note': 'Only available via webpage',
+ 'url': 'http://www.vevo.com/watch/GBUV71600656',
+ 'md5': '67e79210613865b66a47c33baa5e37fe',
+ 'info_dict': {
+ 'id': 'GBUV71600656',
+ 'ext': 'mp4',
+ 'title': 'ABC - Viva Love',
+ 'age_limit': 0,
+ 'timestamp': 1461830400,
+ 'upload_date': '20160428',
+ 'uploader': 'ABC',
+ 'track': 'Viva Love',
+ 'artist': 'ABC',
+ 'genre': 'Pop',
},
+ 'expected_warnings': ['Failed to download video versions info'],
+ }, {
+ # no genres available
+ 'url': 'http://www.vevo.com/watch/INS171400764',
+ 'only_matching': True,
}]
_SMIL_BASE_URL = 'http://smil.lvl3.vevo.com'
_SOURCE_TYPES = {
errnote='Unable to retrieve oauth token')
if 'THIS PAGE IS CURRENTLY UNAVAILABLE IN YOUR REGION' in webpage:
- raise ExtractorError(
- '%s said: This page is currently unavailable in your region.' % self.IE_NAME, expected=True)
+ self.raise_geo_restricted(
+ '%s said: This page is currently unavailable in your region' % self.IE_NAME)
auth_info = self._parse_json(webpage, video_id)
self._api_url_template = self.http_scheme() + '//apiv2.vevo.com/%s?token=' + auth_info['access_token']
- def _call_api(self, path, video_id, note, errnote, fatal=True):
- return self._download_json(self._api_url_template % path, video_id, note, errnote)
+ def _call_api(self, path, *args, **kwargs):
+ return self._download_json(self._api_url_template % path, *args, **kwargs)
def _real_extract(self, url):
video_id = self._match_id(url)
json_url = 'http://api.vevo.com/VideoService/AuthenticateVideo?isrc=%s' % video_id
response = self._download_json(
- json_url, video_id, 'Downloading video info', 'Unable to download info')
+ json_url, video_id, 'Downloading video info',
+ 'Unable to download info', fatal=False) or {}
video_info = response.get('video') or {}
- video_versions = video_info.get('videoVersions')
+ artist = None
+ featured_artist = None
uploader = None
- timestamp = None
view_count = None
formats = []
if not video_info:
- if response.get('statusCode') != 909:
+ try:
+ self._initialize_api(video_id)
+ except ExtractorError:
ytid = response.get('errorInfo', {}).get('ytid')
if ytid:
self.report_warning(
'Video is geoblocked, trying with the YouTube video %s' % ytid)
return self.url_result(ytid, 'Youtube', ytid)
- if 'statusMessage' in response:
- raise ExtractorError('%s said: %s' % (
- self.IE_NAME, response['statusMessage']), expected=True)
- raise ExtractorError('Unable to extract videos')
+ raise
- self._initialize_api(video_id)
video_info = self._call_api(
'video/%s' % video_id, video_id, 'Downloading api video info',
'Failed to download video info')
video_versions = self._call_api(
'video/%s/streams' % video_id, video_id,
'Downloading video versions info',
- 'Failed to download video versions info')
+ 'Failed to download video versions info',
+ fatal=False)
+
+ # Some videos are only available via webpage (e.g.
+ # https://github.com/rg3/youtube-dl/issues/9366)
+ if not video_versions:
+ webpage = self._download_webpage(url, video_id)
+ video_versions = self._extract_json(webpage, video_id, 'streams')[video_id][0]
timestamp = parse_iso8601(video_info.get('releaseDate'))
artists = video_info.get('artists')
if artists:
- uploader = artists[0]['name']
+ artist = uploader = artists[0]['name']
view_count = int_or_none(video_info.get('views', {}).get('total'))
for video_version in video_versions:
scale=1000)
artists = video_info.get('mainArtists')
if artists:
- uploader = artists[0]['artistName']
+ artist = uploader = artists[0]['artistName']
+
+ featured_artists = video_info.get('featuredArtists')
+ if featured_artists:
+ featured_artist = featured_artists[0]['artistName']
smil_parsed = False
for video_version in video_info['videoVersions']:
smil_parsed = True
self._sort_formats(formats)
- title = video_info['title']
+ track = video_info['title']
+ if featured_artist:
+ artist = '%s ft. %s' % (artist, featured_artist)
+ title = '%s - %s' % (artist, track) if artist else track
+
+ genres = video_info.get('genres')
+ genre = (
+ genres[0] if genres and isinstance(genres, list) and
+ isinstance(genres[0], compat_str) else None)
is_explicit = video_info.get('isExplicit')
if is_explicit is True:
'duration': duration,
'view_count': view_count,
'age_limit': age_limit,
+ 'track': track,
+ 'artist': uploader,
+ 'genre': genre,
}
+
+
+class VevoPlaylistIE(VevoBaseIE):
+ _VALID_URL = r'https?://www\.vevo\.com/watch/(?P<kind>playlist|genre)/(?P<id>[^/?#&]+)'
+
+ _TESTS = [{
+ 'url': 'http://www.vevo.com/watch/playlist/dadbf4e7-b99f-4184-9670-6f0e547b6a29',
+ 'info_dict': {
+ 'id': 'dadbf4e7-b99f-4184-9670-6f0e547b6a29',
+ 'title': 'Best-Of: Birdman',
+ },
+ 'playlist_count': 10,
+ }, {
+ 'url': 'http://www.vevo.com/watch/genre/rock',
+ 'info_dict': {
+ 'id': 'rock',
+ 'title': 'Rock',
+ },
+ 'playlist_count': 20,
+ }, {
+ 'url': 'http://www.vevo.com/watch/playlist/dadbf4e7-b99f-4184-9670-6f0e547b6a29?index=0',
+ 'md5': '32dcdfddddf9ec6917fc88ca26d36282',
+ 'info_dict': {
+ 'id': 'USCMV1100073',
+ 'ext': 'mp4',
+ 'title': 'Birdman - Y.U. MAD',
+ 'timestamp': 1323417600,
+ 'upload_date': '20111209',
+ 'uploader': 'Birdman',
+ 'track': 'Y.U. MAD',
+ 'artist': 'Birdman',
+ 'genre': 'Rap/Hip-Hop',
+ },
+ 'expected_warnings': ['Unable to download SMIL file'],
+ }, {
+ 'url': 'http://www.vevo.com/watch/genre/rock?index=0',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ playlist_id = mobj.group('id')
+ playlist_kind = mobj.group('kind')
+
+ webpage = self._download_webpage(url, playlist_id)
+
+ qs = compat_urlparse.parse_qs(compat_urlparse.urlparse(url).query)
+ index = qs.get('index', [None])[0]
+
+ if index:
+ video_id = self._search_regex(
+ r'<meta[^>]+content=(["\'])vevo://video/(?P<id>.+?)\1[^>]*>',
+ webpage, 'video id', default=None, group='id')
+ if video_id:
+ return self.url_result('vevo:%s' % video_id, VevoIE.ie_key())
+
+ playlists = self._extract_json(webpage, playlist_id, '%ss' % playlist_kind)
+
+ playlist = (list(playlists.values())[0]
+ if playlist_kind == 'playlist' else playlists[playlist_id])
+
+ entries = [
+ self.url_result('vevo:%s' % src, VevoIE.ie_key())
+ for src in playlist['isrcs']]
+
+ return self.playlist_result(
+ entries, playlist.get('playlistId') or playlist_id,
+ playlist.get('name'), playlist.get('description'))
_TESTS = [{
'url': 'http://www.vice.com/video/cowboy-capitalists-part-1',
+ 'md5': 'e9d77741f9e42ba583e683cd170660f7',
'info_dict': {
'id': '43cW1mYzpia9IlestBjVpd23Yu3afAfp',
'ext': 'flv',
'title': 'VICE_COWBOYCAPITALISTS_PART01_v1_VICE_WM_1080p.mov',
'duration': 725.983,
},
+ 'add_ie': ['Ooyala'],
}, {
'url': 'http://www.vice.com/video/how-to-hack-a-car',
'md5': '6fb2989a3fed069fb8eab3401fc2d3c9',
'uploader': 'Motherboard',
'upload_date': '20140529',
},
+ 'add_ie': ['Youtube'],
}, {
'url': 'https://news.vice.com/video/experimenting-on-animals-inside-the-monkey-lab',
'only_matching': True,
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import int_or_none
+
+
+class VidioIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?vidio\.com/watch/(?P<id>\d+)-(?P<display_id>[^/?#&]+)'
+ _TESTS = [{
+ 'url': 'http://www.vidio.com/watch/165683-dj_ambred-booyah-live-2015',
+ 'md5': 'cd2801394afc164e9775db6a140b91fe',
+ 'info_dict': {
+ 'id': '165683',
+ 'display_id': 'dj_ambred-booyah-live-2015',
+ 'ext': 'mp4',
+ 'title': 'DJ_AMBRED - Booyah (Live 2015)',
+ 'description': 'md5:27dc15f819b6a78a626490881adbadf8',
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ 'duration': 149,
+ 'like_count': int,
+ },
+ }, {
+ 'url': 'https://www.vidio.com/watch/77949-south-korea-test-fires-missile-that-can-strike-all-of-the-north',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ video_id, display_id = mobj.group('id', 'display_id')
+
+ webpage = self._download_webpage(url, display_id)
+
+ title = self._og_search_title(webpage)
+
+ m3u8_url, duration, thumbnail = [None] * 3
+
+ clips = self._parse_json(
+ self._html_search_regex(
+ r'data-json-clips\s*=\s*(["\'])(?P<data>\[.+?\])\1',
+ webpage, 'video data', default='[]', group='data'),
+ display_id, fatal=False)
+ if clips:
+ clip = clips[0]
+ m3u8_url = clip.get('sources', [{}])[0].get('file')
+ duration = clip.get('clip_duration')
+ thumbnail = clip.get('image')
+
+ m3u8_url = m3u8_url or self._search_regex(
+ r'data(?:-vjs)?-clip-hls-url=(["\'])(?P<url>.+?)\1', webpage, 'hls url')
+ formats = self._extract_m3u8_formats(m3u8_url, display_id, 'mp4', entry_protocol='m3u8_native')
+
+ duration = int_or_none(duration or self._search_regex(
+ r'data-video-duration=(["\'])(?P<duartion>\d+)\1', webpage, 'duration'))
+ thumbnail = thumbnail or self._og_search_thumbnail(webpage)
+
+ like_count = int_or_none(self._search_regex(
+ (r'<span[^>]+data-comment-vote-count=["\'](\d+)',
+ r'<span[^>]+class=["\'].*?\blike(?:__|-)count\b.*?["\'][^>]*>\s*(\d+)'),
+ webpage, 'like count', fatal=False))
+
+ return {
+ 'id': video_id,
+ 'display_id': display_id,
+ 'title': title,
+ 'description': self._og_search_description(webpage),
+ 'thumbnail': thumbnail,
+ 'duration': duration,
+ 'like_count': like_count,
+ 'formats': formats,
+ }
--- /dev/null
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+ ExtractorError,
+ clean_html,
+ determine_ext,
+ int_or_none,
+ js_to_json,
+ parse_duration,
+)
+
+
+class ViewLiftBaseIE(InfoExtractor):
+ _DOMAINS_REGEX = '(?:snagfilms|snagxtreme|funnyforfree|kiddovid|winnersview|monumentalsportsnetwork|vayafilm)\.com|kesari\.tv'
+
+
+class ViewLiftEmbedIE(ViewLiftBaseIE):
+ _VALID_URL = r'https?://(?:(?:www|embed)\.)?(?:%s)/embed/player\?.*\bfilmId=(?P<id>[\da-f-]{36})' % ViewLiftBaseIE._DOMAINS_REGEX
+ _TESTS = [{
+ 'url': 'http://embed.snagfilms.com/embed/player?filmId=74849a00-85a9-11e1-9660-123139220831&w=500',
+ 'md5': '2924e9215c6eff7a55ed35b72276bd93',
+ 'info_dict': {
+ 'id': '74849a00-85a9-11e1-9660-123139220831',
+ 'ext': 'mp4',
+ 'title': '#whilewewatch',
+ }
+ }, {
+ # invalid labels, 360p is better that 480p
+ 'url': 'http://www.snagfilms.com/embed/player?filmId=17ca0950-a74a-11e0-a92a-0026bb61d036',
+ 'md5': '882fca19b9eb27ef865efeeaed376a48',
+ 'info_dict': {
+ 'id': '17ca0950-a74a-11e0-a92a-0026bb61d036',
+ 'ext': 'mp4',
+ 'title': 'Life in Limbo',
+ }
+ }, {
+ 'url': 'http://www.snagfilms.com/embed/player?filmId=0000014c-de2f-d5d6-abcf-ffef58af0017',
+ 'only_matching': True,
+ }]
+
+ @staticmethod
+ def _extract_url(webpage):
+ mobj = re.search(
+ r'<iframe[^>]+?src=(["\'])(?P<url>(?:https?:)?//(?:embed\.)?(?:%s)/embed/player.+?)\1' % ViewLiftBaseIE._DOMAINS_REGEX,
+ webpage)
+ if mobj:
+ return mobj.group('url')
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+
+ webpage = self._download_webpage(url, video_id)
+
+ if '>This film is not playable in your area.<' in webpage:
+ raise ExtractorError(
+ 'Film %s is not playable in your area.' % video_id, expected=True)
+
+ formats = []
+ has_bitrate = False
+ for source in self._parse_json(js_to_json(self._search_regex(
+ r'(?s)sources:\s*(\[.+?\]),', webpage, 'json')), video_id):
+ file_ = source.get('file')
+ if not file_:
+ continue
+ type_ = source.get('type')
+ ext = determine_ext(file_)
+ format_id = source.get('label') or ext
+ if all(v == 'm3u8' or v == 'hls' for v in (type_, ext)):
+ formats.extend(self._extract_m3u8_formats(
+ file_, video_id, 'mp4', m3u8_id='hls'))
+ else:
+ bitrate = int_or_none(self._search_regex(
+ [r'(\d+)kbps', r'_\d{1,2}x\d{1,2}_(\d{3,})\.%s' % ext],
+ file_, 'bitrate', default=None))
+ if not has_bitrate and bitrate:
+ has_bitrate = True
+ height = int_or_none(self._search_regex(
+ r'^(\d+)[pP]$', format_id, 'height', default=None))
+ formats.append({
+ 'url': file_,
+ 'format_id': 'http-%s%s' % (format_id, ('-%dk' % bitrate if bitrate else '')),
+ 'tbr': bitrate,
+ 'height': height,
+ })
+ field_preference = None if has_bitrate else ('height', 'tbr', 'format_id')
+ self._sort_formats(formats, field_preference)
+
+ title = self._search_regex(
+ [r"title\s*:\s*'([^']+)'", r'<title>([^<]+)</title>'],
+ webpage, 'title')
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'formats': formats,
+ }
+
+
+class ViewLiftIE(ViewLiftBaseIE):
+ _VALID_URL = r'https?://(?:www\.)?(?P<domain>%s)/(?:films/title|show|(?:news/)?videos?)/(?P<id>[^?#]+)' % ViewLiftBaseIE._DOMAINS_REGEX
+ _TESTS = [{
+ 'url': 'http://www.snagfilms.com/films/title/lost_for_life',
+ 'md5': '19844f897b35af219773fd63bdec2942',
+ 'info_dict': {
+ 'id': '0000014c-de2f-d5d6-abcf-ffef58af0017',
+ 'display_id': 'lost_for_life',
+ 'ext': 'mp4',
+ 'title': 'Lost for Life',
+ 'description': 'md5:fbdacc8bb6b455e464aaf98bc02e1c82',
+ 'thumbnail': 're:^https?://.*\.jpg',
+ 'duration': 4489,
+ 'categories': ['Documentary', 'Crime', 'Award Winning', 'Festivals']
+ }
+ }, {
+ 'url': 'http://www.snagfilms.com/show/the_world_cut_project/india',
+ 'md5': 'e6292e5b837642bbda82d7f8bf3fbdfd',
+ 'info_dict': {
+ 'id': '00000145-d75c-d96e-a9c7-ff5c67b20000',
+ 'display_id': 'the_world_cut_project/india',
+ 'ext': 'mp4',
+ 'title': 'India',
+ 'description': 'md5:5c168c5a8f4719c146aad2e0dfac6f5f',
+ 'thumbnail': 're:^https?://.*\.jpg',
+ 'duration': 979,
+ 'categories': ['Documentary', 'Sports', 'Politics']
+ }
+ }, {
+ # Film is not playable in your area.
+ 'url': 'http://www.snagfilms.com/films/title/inside_mecca',
+ 'only_matching': True,
+ }, {
+ # Film is not available.
+ 'url': 'http://www.snagfilms.com/show/augie_alone/flirting',
+ 'only_matching': True,
+ }, {
+ 'url': 'http://www.winnersview.com/videos/the-good-son',
+ 'only_matching': True,
+ }, {
+ 'url': 'http://www.kesari.tv/news/video/1461919076414',
+ 'only_matching': True,
+ }, {
+ # Was once Kaltura embed
+ 'url': 'https://www.monumentalsportsnetwork.com/videos/john-carlson-postgame-2-25-15',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ domain, display_id = re.match(self._VALID_URL, url).groups()
+
+ webpage = self._download_webpage(url, display_id)
+
+ if ">Sorry, the Film you're looking for is not available.<" in webpage:
+ raise ExtractorError(
+ 'Film %s is not available.' % display_id, expected=True)
+
+ film_id = self._search_regex(r'filmId=([\da-f-]{36})"', webpage, 'film id')
+
+ snag = self._parse_json(
+ self._search_regex(
+ 'Snag\.page\.data\s*=\s*(\[.+?\]);', webpage, 'snag'),
+ display_id)
+
+ for item in snag:
+ if item.get('data', {}).get('film', {}).get('id') == film_id:
+ data = item['data']['film']
+ title = data['title']
+ description = clean_html(data.get('synopsis'))
+ thumbnail = data.get('image')
+ duration = int_or_none(data.get('duration') or data.get('runtime'))
+ categories = [
+ category['title'] for category in data.get('categories', [])
+ if category.get('title')]
+ break
+ else:
+ title = self._search_regex(
+ r'itemprop="title">([^<]+)<', webpage, 'title')
+ description = self._html_search_regex(
+ r'(?s)<div itemprop="description" class="film-synopsis-inner ">(.+?)</div>',
+ webpage, 'description', default=None) or self._og_search_description(webpage)
+ thumbnail = self._og_search_thumbnail(webpage)
+ duration = parse_duration(self._search_regex(
+ r'<span itemprop="duration" class="film-duration strong">([^<]+)<',
+ webpage, 'duration', fatal=False))
+ categories = re.findall(r'<a href="/movies/[^"]+">([^<]+)</a>', webpage)
+
+ return {
+ '_type': 'url_transparent',
+ 'url': 'http://%s/embed/player?filmId=%s' % (domain, film_id),
+ 'id': film_id,
+ 'display_id': display_id,
+ 'title': title,
+ 'description': description,
+ 'thumbnail': thumbnail,
+ 'duration': duration,
+ 'categories': categories,
+ 'ie_key': 'ViewLiftEmbed',
+ }
# coding: utf-8
from __future__ import unicode_literals
+import re
+
from .common import InfoExtractor
from ..compat import (
compat_HTTPError,
- compat_urllib_parse,
compat_urllib_parse_unquote,
)
from ..utils import (
parse_iso8601,
sanitized_Request,
HEADRequest,
+ url_basename,
)
_ACCEPT_HEADER = 'application/json, text/javascript, */*; q=0.01'
- def _download_json(self, url, video_id, note='Downloading JSON metadata', fatal=True):
+ def _download_json(self, url, video_id, note='Downloading JSON metadata', fatal=True, query={}):
request = sanitized_Request(url)
request.add_header('Accept', self._ACCEPT_HEADER)
request.add_header('Auth-token', self._AUTH_TOKEN)
- return super(ViewsterIE, self)._download_json(request, video_id, note, fatal=fatal)
+ return super(ViewsterIE, self)._download_json(request, video_id, note, fatal=fatal, query=query)
def _real_extract(self, url):
video_id = self._match_id(url)
return self.playlist_result(entries, video_id, title, description)
formats = []
- for media_type in ('application/f4m+xml', 'application/x-mpegURL', 'video/mp4'):
- media = self._download_json(
- 'https://public-api.viewster.com/movies/%s/video?mediaType=%s'
- % (entry_id, compat_urllib_parse.quote(media_type)),
- video_id, 'Downloading %s JSON' % media_type, fatal=False)
- if not media:
- continue
- video_url = media.get('Uri')
- if not video_url:
- continue
- ext = determine_ext(video_url)
- if ext == 'f4m':
- video_url += '&' if '?' in video_url else '?'
- video_url += 'hdcore=3.2.0&plugin=flowplayer-3.2.0.1'
- formats.extend(self._extract_f4m_formats(
- video_url, video_id, f4m_id='hds'))
- elif ext == 'm3u8':
- m3u8_formats = self._extract_m3u8_formats(
- video_url, video_id, 'mp4', m3u8_id='hls',
- fatal=False) # m3u8 sometimes fail
- if m3u8_formats:
- formats.extend(m3u8_formats)
- else:
- format_id = media.get('Bitrate')
- f = {
- 'url': video_url,
- 'format_id': 'mp4-%s' % format_id,
- 'height': int_or_none(media.get('Height')),
- 'width': int_or_none(media.get('Width')),
- 'preference': 1,
- }
- if format_id and not f['height']:
- f['height'] = int_or_none(self._search_regex(
- r'^(\d+)[pP]$', format_id, 'height', default=None))
- formats.append(f)
-
- if not formats and not info.get('LanguageSets') and not info.get('VODSettings'):
+ for language_set in info.get('LanguageSets', []):
+ manifest_url = None
+ m3u8_formats = []
+ audio = language_set.get('Audio') or ''
+ subtitle = language_set.get('Subtitle') or ''
+ base_format_id = audio
+ if subtitle:
+ base_format_id += '-%s' % subtitle
+
+ def concat(suffix, sep='-'):
+ return (base_format_id + '%s%s' % (sep, suffix)) if base_format_id else suffix
+
+ for media_type in ('application/f4m+xml', 'application/x-mpegURL', 'video/mp4'):
+ media = self._download_json(
+ 'https://public-api.viewster.com/movies/%s/video' % entry_id,
+ video_id, 'Downloading %s JSON' % concat(media_type, ' '), fatal=False, query={
+ 'mediaType': media_type,
+ 'language': audio,
+ 'subtitle': subtitle,
+ })
+ if not media:
+ continue
+ video_url = media.get('Uri')
+ if not video_url:
+ continue
+ ext = determine_ext(video_url)
+ if ext == 'f4m':
+ manifest_url = video_url
+ video_url += '&' if '?' in video_url else '?'
+ video_url += 'hdcore=3.2.0&plugin=flowplayer-3.2.0.1'
+ formats.extend(self._extract_f4m_formats(
+ video_url, video_id, f4m_id=concat('hds')))
+ elif ext == 'm3u8':
+ manifest_url = video_url
+ m3u8_formats = self._extract_m3u8_formats(
+ video_url, video_id, 'mp4', m3u8_id=concat('hls'),
+ fatal=False) # m3u8 sometimes fail
+ if m3u8_formats:
+ formats.extend(m3u8_formats)
+ else:
+ qualities_basename = self._search_regex(
+ '/([^/]+)\.csmil/',
+ manifest_url, 'qualities basename', default=None)
+ if not qualities_basename:
+ continue
+ QUALITIES_RE = r'((,\d+k)+,?)'
+ qualities = self._search_regex(
+ QUALITIES_RE, qualities_basename,
+ 'qualities', default=None)
+ if not qualities:
+ continue
+ qualities = list(map(lambda q: int(q[:-1]), qualities.strip(',').split(',')))
+ qualities.sort()
+ http_template = re.sub(QUALITIES_RE, r'%dk', qualities_basename)
+ http_url_basename = url_basename(video_url)
+ if m3u8_formats:
+ self._sort_formats(m3u8_formats)
+ m3u8_formats = list(filter(
+ lambda f: f.get('vcodec') != 'none' and f.get('resolution') != 'multiple',
+ m3u8_formats))
+ if len(qualities) == len(m3u8_formats):
+ for q, m3u8_format in zip(qualities, m3u8_formats):
+ f = m3u8_format.copy()
+ f.update({
+ 'url': video_url.replace(http_url_basename, http_template % q),
+ 'format_id': f['format_id'].replace('hls', 'http'),
+ 'protocol': 'http',
+ })
+ formats.append(f)
+ else:
+ for q in qualities:
+ formats.append({
+ 'url': video_url.replace(http_url_basename, http_template % q),
+ 'ext': 'mp4',
+ 'format_id': 'http-%d' % q,
+ 'tbr': q,
+ })
+
+ if not formats and not info.get('VODSettings'):
self.raise_geo_restricted()
self._sort_formats(formats)
_VALID_URL = r'''(?x)
https?://
(?:
- (?:m\.)?vk\.com/video_ext\.php\?.*?\boid=(?P<oid>-?\d+).*?\bid=(?P<id>\d+)|
+ (?:
+ (?:m\.)?vk\.com/video_|
+ (?:www\.)?daxab.com/
+ )
+ ext\.php\?(?P<embed_query>.*?\boid=(?P<oid>-?\d+).*?\bid=(?P<id>\d+).*)|
(?:
(?:m\.)?vk\.com/(?:.+?\?.*?z=)?video|
- (?:www\.)?biqle\.ru/watch/
+ (?:www\.)?daxab.com/embed/
)
- (?P<videoid>[^s].*?)(?:\?(?:.*\blist=(?P<list_id>[\da-f]+))?|%2F|$)
+ (?P<videoid>-?\d+_\d+)(?:.*\blist=(?P<list_id>[\da-f]+))?
)
'''
_NETRC_MACHINE = 'vk'
'duration': 101,
'upload_date': '20120730',
'view_count': int,
- }
+ },
+ 'skip': 'This video has been removed from public access.',
},
{
# VIDEO NOW REMOVED
'id': 'V3K4mi0SYkc',
'ext': 'webm',
'title': "DSWD Awards 'Children's Joy Foundation, Inc.' Certificate of Registration and License to Operate",
- 'description': 'md5:bf9c26cfa4acdfb146362682edd3827a',
+ 'description': 'md5:d9903938abdc74c738af77f527ca0596',
'duration': 178,
'upload_date': '20130116',
'uploader': "Children's Joy Foundation",
'url': 'https://vk.com/video205387401_164765225',
'only_matching': True,
},
- {
- # vk wrapper
- 'url': 'http://www.biqle.ru/watch/847655_160197695',
- 'only_matching': True,
- },
{
# pladform embed
'url': 'https://vk.com/video-76116461_171554880',
mobj = re.match(self._VALID_URL, url)
video_id = mobj.group('videoid')
- if not video_id:
+ if video_id:
+ info_url = 'https://vk.com/al_video.php?act=show&al=1&module=video&video=%s' % video_id
+ # Some videos (removed?) can only be downloaded with list id specified
+ list_id = mobj.group('list_id')
+ if list_id:
+ info_url += '&list=%s' % list_id
+ else:
+ info_url = 'http://vk.com/video_ext.php?' + mobj.group('embed_query')
video_id = '%s_%s' % (mobj.group('oid'), mobj.group('id'))
- info_url = 'https://vk.com/al_video.php?act=show&al=1&module=video&video=%s' % video_id
-
- # Some videos (removed?) can only be downloaded with list id specified
- list_id = mobj.group('list_id')
- if list_id:
- info_url += '&list=%s' % list_id
-
info_page = self._download_webpage(info_url, video_id)
error_message = self._html_search_regex(
- r'(?s)<!><div[^>]+class="video_layer_message"[^>]*>(.+?)</div>',
+ [r'(?s)<!><div[^>]+class="video_layer_message"[^>]*>(.+?)</div>',
+ r'(?s)<div[^>]+id="video_ext_msg"[^>]*>(.+?)</div>'],
info_page, 'error message', default=None)
if error_message:
raise ExtractorError(error_message, expected=True)
view_count = None
views = self._html_search_regex(
r'"mv_views_count_number"[^>]*>(.+?\bviews?)<',
- info_page, 'view count', fatal=False)
+ info_page, 'view count', default=None)
if views:
view_count = str_to_int(self._search_regex(
r'([\d,.]+)', views, 'view count', fatal=False))
formats = []
for k, v in data.items():
- if not k.startswith('url') and k != 'extra_data' or not v:
+ if not k.startswith('url') and not k.startswith('cache') and k != 'extra_data' or not v:
continue
height = int_or_none(self._search_regex(
- r'^url(\d+)', k, 'height', default=None))
+ r'^(?:url|cache)(\d+)', k, 'height', default=None))
formats.append({
'format_id': k,
'url': v,
# coding: utf-8
from __future__ import unicode_literals
+import re
+
from .common import InfoExtractor
from ..utils import (
dict_get,
+ ExtractorError,
float_or_none,
int_or_none,
)
'info_dict': {
'id': '1326',
'ext': 'mp4',
- 'title': "[V] Girl's Day's Broadcast",
+ 'title': "[V LIVE] Girl's Day's Broadcast",
'creator': "Girl's Day",
'view_count': int,
},
webpage = self._download_webpage(
'http://www.vlive.tv/video/%s' % video_id, video_id)
- long_video_id = self._search_regex(
- r'vlive\.tv\.video\.ajax\.request\.handler\.init\(\s*"[0-9]+"\s*,\s*"[^"]*"\s*,\s*"([^"]+)"',
- webpage, 'long video id')
+ video_params = self._search_regex(
+ r'\bvlive\.video\.init\(([^)]+)\)',
+ webpage, 'video params')
+ status, _, _, live_params, long_video_id, key = re.split(
+ r'"\s*,\s*"', video_params)[2:8]
+
+ if status == 'LIVE_ON_AIR' or status == 'BIG_EVENT_ON_AIR':
+ live_params = self._parse_json('"%s"' % live_params, video_id)
+ live_params = self._parse_json(live_params, video_id)
+ return self._live(video_id, webpage, live_params)
+ elif status == 'VOD_ON_AIR' or status == 'BIG_EVENT_INTRO':
+ if long_video_id and key:
+ return self._replay(video_id, webpage, long_video_id, key)
+ else:
+ status = 'COMING_SOON'
- key = self._search_regex(
- r'vlive\.tv\.video\.ajax\.request\.handler\.init\(\s*"[0-9]+"\s*,\s*"[^"]*"\s*,\s*"[^"]+"\s*,\s*"([^"]+)"',
- webpage, 'key')
+ if status == 'LIVE_END':
+ raise ExtractorError('Uploading for replay. Please wait...',
+ expected=True)
+ elif status == 'COMING_SOON':
+ raise ExtractorError('Coming soon!', expected=True)
+ elif status == 'CANCELED':
+ raise ExtractorError('We are sorry, '
+ 'but the live broadcast has been canceled.',
+ expected=True)
+ else:
+ raise ExtractorError('Unknown status %s' % status)
+ def _get_common_fields(self, webpage):
title = self._og_search_title(webpage)
+ creator = self._html_search_regex(
+ r'<div[^>]+class="info_area"[^>]*>\s*<a\s+[^>]*>([^<]+)',
+ webpage, 'creator', fatal=False)
+ thumbnail = self._og_search_thumbnail(webpage)
+ return {
+ 'title': title,
+ 'creator': creator,
+ 'thumbnail': thumbnail,
+ }
+
+ def _live(self, video_id, webpage, live_params):
+ formats = []
+ for vid in live_params.get('resolutions', []):
+ formats.extend(self._extract_m3u8_formats(
+ vid['cdnUrl'], video_id, 'mp4',
+ m3u8_id=vid.get('name'),
+ fatal=False, live=True))
+ self._sort_formats(formats)
+
+ return dict(self._get_common_fields(webpage),
+ id=video_id,
+ formats=formats,
+ is_live=True)
+ def _replay(self, video_id, webpage, long_video_id, key):
playinfo = self._download_json(
'http://global.apis.naver.com/rmcnmv/rmcnmv/vod_play_videoInfo.json?%s'
% compat_urllib_parse_urlencode({
} for vid in playinfo.get('videos', {}).get('list', []) if vid.get('source')]
self._sort_formats(formats)
- thumbnail = self._og_search_thumbnail(webpage)
- creator = self._html_search_regex(
- r'<div[^>]+class="info_area"[^>]*>\s*<a\s+[^>]*>([^<]+)',
- webpage, 'creator', fatal=False)
-
view_count = int_or_none(playinfo.get('meta', {}).get('count'))
subtitles = {}
'ext': 'vtt',
'url': caption['source']}]
- return {
- 'id': video_id,
- 'title': title,
- 'creator': creator,
- 'thumbnail': thumbnail,
- 'view_count': view_count,
- 'formats': formats,
- 'subtitles': subtitles,
- }
+ return dict(self._get_common_fields(webpage),
+ id=video_id,
+ formats=formats,
+ view_count=view_count,
+ subtitles=subtitles)
'ext': 'mp4',
'title': 'Google\'s new material design direction',
'description': 'md5:2f44f74c4d14a1f800ea73e1c6832ad2',
- }
+ },
+ 'add_ie': ['Ooyala'],
}, {
# data-ooyala-id
'url': 'http://www.theverge.com/2014/10/21/7025853/google-nexus-6-hands-on-photos-video-android-phablet',
'ext': 'mp4',
'title': 'The Nexus 6: hands-on with Google\'s phablet',
'description': 'md5:87a51fe95ff8cea8b5bdb9ac7ae6a6af',
- }
+ },
+ 'add_ie': ['Ooyala'],
}, {
# volume embed
'url': 'http://www.vox.com/2016/3/31/11336640/mississippi-lgbt-religious-freedom-bill',
'ext': 'mp4',
'title': 'The new frontier of LGBTQ civil rights, explained',
'description': 'md5:0dc58e94a465cbe91d02950f770eb93f',
- }
+ },
+ 'add_ie': ['Ooyala'],
}, {
# youtube embed
'url': 'http://www.vox.com/2016/3/24/11291692/robot-dance',
'upload_date': '20160324',
'uploader_id': 'voxdotcom',
'uploader': 'Vox',
- }
+ },
+ 'add_ie': ['Youtube'],
}, {
# SBN.VideoLinkset.entryGroup multiple ooyala embeds
'url': 'http://www.sbnation.com/college-football-recruiting/2015/2/3/7970291/national-signing-day-rationalizations-itll-be-ok-itll-be-ok',
volume_webpage = self._download_webpage(
'http://volume.vox-cdn.com/embed/%s' % volume_uuid, volume_uuid)
video_data = self._parse_json(self._search_regex(
- r'Volume\.createVideo\(({.+})\s*,\s*{.*}\);', volume_webpage, 'video data'), volume_uuid)
+ r'Volume\.createVideo\(({.+})\s*,\s*{.*}\s*,\s*\[.*\]\s*,\s*{.*}\);', volume_webpage, 'video data'), volume_uuid)
for provider_video_type in ('ooyala', 'youtube'):
provider_video_id = video_data.get('%s_id' % provider_video_type)
if provider_video_id:
class WashingtonPostIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?washingtonpost\.com/.*?/(?P<id>[^/]+)/(?:$|[?#])'
+ IE_NAME = 'washingtonpost'
+ _VALID_URL = r'(?:washingtonpost:|https?://(?:www\.)?washingtonpost\.com/video/(?:[^/]+/)*)(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
+ _TEST = {
+ 'url': 'https://www.washingtonpost.com/video/c/video/480ba4ee-1ec7-11e6-82c2-a7dcb313287d',
+ 'md5': '6f537e1334b714eb15f9563bd4b9cdfa',
+ 'info_dict': {
+ 'id': '480ba4ee-1ec7-11e6-82c2-a7dcb313287d',
+ 'ext': 'mp4',
+ 'title': 'Egypt finds belongings, debris from plane crash',
+ 'description': 'md5:a17ceee432f215a5371388c1f680bd86',
+ 'upload_date': '20160520',
+ 'uploader': 'Reuters',
+ 'timestamp': 1463778452,
+ },
+ }
+
+ def _real_extract(self, url):
+ video_id = self._match_id(url)
+ video_data = self._download_json(
+ 'http://www.washingtonpost.com/posttv/c/videojson/%s?resType=jsonp' % video_id,
+ video_id, transform_source=strip_jsonp)[0]['contentConfig']
+ title = video_data['title']
+
+ urls = []
+ formats = []
+ for s in video_data.get('streams', []):
+ s_url = s.get('url')
+ if not s_url or s_url in urls:
+ continue
+ urls.append(s_url)
+ video_type = s.get('type')
+ if video_type == 'smil':
+ continue
+ elif video_type in ('ts', 'hls') and ('_master.m3u8' in s_url or '_mobile.m3u8' in s_url):
+ m3u8_formats = self._extract_m3u8_formats(
+ s_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False)
+ for m3u8_format in m3u8_formats:
+ width = m3u8_format.get('width')
+ if not width:
+ continue
+ vbr = self._search_regex(
+ r'%d_%d_(\d+)' % (width, m3u8_format['height']), m3u8_format['url'], 'vbr', default=None)
+ if vbr:
+ m3u8_format.update({
+ 'vbr': int_or_none(vbr),
+ })
+ formats.extend(m3u8_formats)
+ else:
+ width = int_or_none(s.get('width'))
+ vbr = int_or_none(s.get('bitrate'))
+ has_width = width != 0
+ formats.append({
+ 'format_id': (
+ '%s-%d-%d' % (video_type, width, vbr)
+ if width
+ else video_type),
+ 'vbr': vbr if has_width else None,
+ 'width': width,
+ 'height': int_or_none(s.get('height')),
+ 'acodec': s.get('audioCodec'),
+ 'vcodec': s.get('videoCodec') if has_width else 'none',
+ 'filesize': int_or_none(s.get('fileSize')),
+ 'url': s_url,
+ 'ext': 'mp4',
+ 'protocol': 'm3u8_native' if video_type in ('ts', 'hls') else None,
+ })
+ source_media_url = video_data.get('sourceMediaURL')
+ if source_media_url:
+ formats.append({
+ 'format_id': 'source_media',
+ 'url': source_media_url,
+ })
+ self._sort_formats(
+ formats, ('width', 'height', 'vbr', 'filesize', 'tbr', 'format_id'))
+
+ return {
+ 'id': video_id,
+ 'title': title,
+ 'description': video_data.get('blurb'),
+ 'uploader': video_data.get('credits', {}).get('source'),
+ 'formats': formats,
+ 'duration': int_or_none(video_data.get('videoDuration'), 100),
+ 'timestamp': int_or_none(
+ video_data.get('dateConfig', {}).get('dateFirstPublished'), 1000),
+ }
+
+
+class WashingtonPostArticleIE(InfoExtractor):
+ IE_NAME = 'washingtonpost:article'
+ _VALID_URL = r'https?://(?:www\.)?washingtonpost\.com/(?:[^/]+/)*(?P<id>[^/?#]+)'
_TESTS = [{
'url': 'http://www.washingtonpost.com/sf/national/2014/03/22/sinkhole-of-bureaucracy/',
'info_dict': {
}]
}]
+ @classmethod
+ def suitable(cls, url):
+ return False if WashingtonPostIE.suitable(url) else super(WashingtonPostArticleIE, cls).suitable(url)
+
def _real_extract(self, url):
page_id = self._match_id(url)
webpage = self._download_webpage(url, page_id)
<div\s+class="posttv-video-embed[^>]*?data-uuid=|
data-video-uuid=
)"([^"]+)"''', webpage)
- entries = []
- for i, uuid in enumerate(uuids, start=1):
- vinfo_all = self._download_json(
- 'http://www.washingtonpost.com/posttv/c/videojson/%s?resType=jsonp' % uuid,
- page_id,
- transform_source=strip_jsonp,
- note='Downloading information of video %d/%d' % (i, len(uuids))
- )
- vinfo = vinfo_all[0]['contentConfig']
- uploader = vinfo.get('credits', {}).get('source')
- timestamp = int_or_none(
- vinfo.get('dateConfig', {}).get('dateFirstPublished'), 1000)
-
- formats = [{
- 'format_id': (
- '%s-%s-%s' % (s.get('type'), s.get('width'), s.get('bitrate'))
- if s.get('width')
- else s.get('type')),
- 'vbr': s.get('bitrate') if s.get('width') != 0 else None,
- 'width': s.get('width'),
- 'height': s.get('height'),
- 'acodec': s.get('audioCodec'),
- 'vcodec': s.get('videoCodec') if s.get('width') != 0 else 'none',
- 'filesize': s.get('fileSize'),
- 'url': s.get('url'),
- 'ext': 'mp4',
- 'preference': -100 if s.get('type') == 'smil' else None,
- 'protocol': {
- 'MP4': 'http',
- 'F4F': 'f4m',
- }.get(s.get('type')),
- } for s in vinfo.get('streams', [])]
- source_media_url = vinfo.get('sourceMediaURL')
- if source_media_url:
- formats.append({
- 'format_id': 'source_media',
- 'url': source_media_url,
- })
- self._sort_formats(formats)
- entries.append({
- 'id': uuid,
- 'title': vinfo['title'],
- 'description': vinfo.get('blurb'),
- 'uploader': uploader,
- 'formats': formats,
- 'duration': int_or_none(vinfo.get('videoDuration'), 100),
- 'timestamp': timestamp,
- })
+ entries = [self.url_result('washingtonpost:%s' % uuid, 'WashingtonPost', uuid) for uuid in uuids]
return {
'_type': 'playlist',
from __future__ import unicode_literals
import re
-import hashlib
from .common import InfoExtractor
+from ..compat import compat_str
from ..utils import (
ExtractorError,
unified_strdate,
+ HEADRequest,
+ float_or_none,
)
class WatIE(InfoExtractor):
- _VALID_URL = r'(?:wat:(?P<real_id>\d{8})|https?://www\.wat\.tv/video/(?P<display_id>.*)-(?P<short_id>.*?)_.*?\.html)'
+ _VALID_URL = r'(?:wat:|https?://(?:www\.)?wat\.tv/video/.*-)(?P<id>[0-9a-z]+)'
IE_NAME = 'wat.tv'
_TESTS = [
{
'url': 'http://www.wat.tv/video/soupe-figues-l-orange-aux-epices-6z1uz_2hvf7_.html',
- 'md5': 'ce70e9223945ed26a8056d413ca55dc9',
+ 'md5': '83d882d9de5c9d97f0bb2c6273cde56a',
'info_dict': {
'id': '11713067',
- 'display_id': 'soupe-figues-l-orange-aux-epices',
'ext': 'mp4',
'title': 'Soupe de figues à l\'orange et aux épices',
'description': 'Retrouvez l\'émission "Petits plats en équilibre", diffusée le 18 août 2014.',
'md5': 'fbc84e4378165278e743956d9c1bf16b',
'info_dict': {
'id': '11713075',
- 'display_id': 'gregory-lemarchal-voix-ange',
'ext': 'mp4',
'title': 'Grégory Lemarchal, une voix d\'ange depuis 10 ans (1/3)',
'description': 'md5:b7a849cf16a2b733d9cd10c52906dee3',
},
]
- def download_video_info(self, real_id):
- # 'contentv4' is used in the website, but it also returns the related
- # videos, we don't need them
- info = self._download_json('http://www.wat.tv/interface/contentv3/' + real_id, real_id)
- return info['media']
-
def _real_extract(self, url):
- def real_id_for_chapter(chapter):
- return chapter['tc_start'].split('-')[0]
- mobj = re.match(self._VALID_URL, url)
- display_id = mobj.group('display_id')
- real_id = mobj.group('real_id')
- if not real_id:
- short_id = mobj.group('short_id')
- webpage = self._download_webpage(url, display_id or short_id)
- real_id = self._search_regex(r'xtpage = ".*-(.*?)";', webpage, 'real id')
+ video_id = self._match_id(url)
+ video_id = video_id if video_id.isdigit() and len(video_id) > 6 else compat_str(int(video_id, 36))
- video_info = self.download_video_info(real_id)
+ # 'contentv4' is used in the website, but it also returns the related
+ # videos, we don't need them
+ video_info = self._download_json(
+ 'http://www.wat.tv/interface/contentv3/' + video_id, video_id)['media']
error_desc = video_info.get('error_desc')
if error_desc:
raise ExtractorError(
'%s returned error: %s' % (self.IE_NAME, error_desc), expected=True)
- geo_list = video_info.get('geoList')
- country = geo_list[0] if geo_list else ''
-
chapters = video_info['chapters']
first_chapter = chapters[0]
- files = video_info['files']
- first_file = files[0]
- if real_id_for_chapter(first_chapter) != real_id:
- self.to_screen('Multipart video detected')
- chapter_urls = []
- for chapter in chapters:
- chapter_id = real_id_for_chapter(chapter)
- # Yes, when we this chapter is processed by WatIE,
- # it will download the info again
- chapter_info = self.download_video_info(chapter_id)
- chapter_urls.append(chapter_info['url'])
- entries = [self.url_result(chapter_url) for chapter_url in chapter_urls]
- return self.playlist_result(entries, real_id, video_info['title'])
+ def video_id_for_chapter(chapter):
+ return chapter['tc_start'].split('-')[0]
- upload_date = None
- if 'date_diffusion' in first_chapter:
- upload_date = unified_strdate(first_chapter['date_diffusion'])
+ if video_id_for_chapter(first_chapter) != video_id:
+ self.to_screen('Multipart video detected')
+ entries = [self.url_result('wat:%s' % video_id_for_chapter(chapter)) for chapter in chapters]
+ return self.playlist_result(entries, video_id, video_info['title'])
# Otherwise we can continue and extract just one part, we have to use
- # the short id for getting the video url
-
- formats = [{
- 'url': 'http://wat.tv/get/android5/%s.mp4' % real_id,
- 'format_id': 'Mobile',
- }]
-
- fmts = [('SD', 'web')]
- if first_file.get('hasHD'):
- fmts.append(('HD', 'webhd'))
-
- def compute_token(param):
- timestamp = '%08x' % int(self._download_webpage(
- 'http://www.wat.tv/servertime', real_id,
- 'Downloading server time').split('|')[0])
- magic = '9b673b13fa4682ed14c3cfa5af5310274b514c4133e9b3a81e6e3aba009l2564'
- return '%s/%s' % (hashlib.md5((magic + param + timestamp).encode('ascii')).hexdigest(), timestamp)
-
- for fmt in fmts:
- webid = '/%s/%s' % (fmt[1], real_id)
- video_url = self._download_webpage(
- 'http://www.wat.tv/get%s?token=%s&getURL=1&country=%s' % (webid, compute_token(webid), country),
- real_id,
- 'Downloading %s video URL' % fmt[0],
- 'Failed to download %s video URL' % fmt[0],
- False)
- if not video_url:
+ # the video id for getting the video url
+
+ date_diffusion = first_chapter.get('date_diffusion')
+ upload_date = unified_strdate(date_diffusion) if date_diffusion else None
+
+ def extract_url(path_template, url_type):
+ req_url = 'http://www.wat.tv/get/%s' % (path_template % video_id)
+ head = self._request_webpage(HEADRequest(req_url), video_id, 'Extracting %s url' % url_type)
+ red_url = head.geturl()
+ if req_url == red_url:
+ raise ExtractorError(
+ '%s said: Sorry, this video is not available from your country.' % self.IE_NAME,
+ expected=True)
+ return red_url
+
+ m3u8_url = extract_url('ipad/%s.m3u8', 'm3u8')
+ http_url = extract_url('android5/%s.mp4', 'http')
+
+ formats = []
+ m3u8_formats = self._extract_m3u8_formats(
+ m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls')
+ formats.extend(m3u8_formats)
+ formats.extend(self._extract_f4m_formats(
+ m3u8_url.replace('ios.', 'web.').replace('.m3u8', '.f4m'),
+ video_id, f4m_id='hds', fatal=False))
+ for m3u8_format in m3u8_formats:
+ mobj = re.search(
+ r'audio.*?%3D(\d+)(?:-video.*?%3D(\d+))?', m3u8_format['url'])
+ if not mobj:
continue
- formats.append({
- 'url': video_url,
- 'ext': 'mp4',
- 'format_id': fmt[0],
+ abr, vbr = mobj.groups()
+ abr, vbr = float_or_none(abr, 1000), float_or_none(vbr, 1000)
+ m3u8_format.update({
+ 'vbr': vbr,
+ 'abr': abr,
+ })
+ if not vbr or not abr:
+ continue
+ f = m3u8_format.copy()
+ f.update({
+ 'url': re.sub(r'%s-\d+00-\d+' % video_id, '%s-%d00-%d' % (video_id, round(vbr / 100), round(abr)), http_url),
+ 'format_id': f['format_id'].replace('hls', 'http'),
+ 'protocol': 'http',
})
+ formats.append(f)
+ self._sort_formats(formats)
return {
- 'id': real_id,
- 'display_id': display_id,
+ 'id': video_id,
'title': first_chapter['title'],
'thumbnail': first_chapter['preview'],
'description': first_chapter['description'],
'view_count': video_info['views'],
'upload_date': upload_date,
- 'duration': first_file['duration'],
+ 'duration': video_info['files'][0]['duration'],
'formats': formats,
}
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+ unified_strdate,
+ parse_duration,
+ int_or_none,
+)
+
+
+class WatchIndianPornIE(InfoExtractor):
+ IE_DESC = 'Watch Indian Porn'
+ _VALID_URL = r'https?://(?:www\.)?watchindianporn\.net/(?:[^/]+/)*video/(?P<display_id>[^/]+)-(?P<id>[a-zA-Z0-9]+)\.html'
+ _TEST = {
+ 'url': 'http://www.watchindianporn.net/video/hot-milf-from-kerala-shows-off-her-gorgeous-large-breasts-on-camera-RZa2avywNPa.html',
+ 'md5': '249589a164dde236ec65832bfce17440',
+ 'info_dict': {
+ 'id': 'RZa2avywNPa',
+ 'display_id': 'hot-milf-from-kerala-shows-off-her-gorgeous-large-breasts-on-camera',
+ 'ext': 'mp4',
+ 'title': 'Hot milf from kerala shows off her gorgeous large breasts on camera',
+ 'thumbnail': 're:^https?://.*\.jpg$',
+ 'uploader': 'LoveJay',
+ 'upload_date': '20160428',
+ 'duration': 226,
+ 'view_count': int,
+ 'comment_count': int,
+ 'categories': list,
+ 'age_limit': 18,
+ }
+ }
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ video_id = mobj.group('id')
+ display_id = mobj.group('display_id')
+
+ webpage = self._download_webpage(url, display_id)
+
+ video_url = self._html_search_regex(
+ r"url: escape\('([^']+)'\)", webpage, 'url')
+
+ title = self._html_search_regex(
+ r'<h2 class="he2"><span>(.*?)</span>',
+ webpage, 'title')
+ thumbnail = self._html_search_regex(
+ r'<span id="container"><img\s+src="([^"]+)"',
+ webpage, 'thumbnail', fatal=False)
+
+ uploader = self._html_search_regex(
+ r'class="aupa">\s*(.*?)</a>',
+ webpage, 'uploader')
+ upload_date = unified_strdate(self._html_search_regex(
+ r'Added: <strong>(.+?)</strong>', webpage, 'upload date', fatal=False))
+
+ duration = parse_duration(self._search_regex(
+ r'<td>Time:\s*</td>\s*<td align="right"><span>\s*(.+?)\s*</span>',
+ webpage, 'duration', fatal=False))
+
+ view_count = int_or_none(self._search_regex(
+ r'<td>Views:\s*</td>\s*<td align="right"><span>\s*(\d+)\s*</span>',
+ webpage, 'view count', fatal=False))
+ comment_count = int_or_none(self._search_regex(
+ r'<td>Comments:\s*</td>\s*<td align="right"><span>\s*(\d+)\s*</span>',
+ webpage, 'comment count', fatal=False))
+
+ categories = re.findall(
+ r'<a href="[^"]+/search/video/desi"><span>([^<]+)</span></a>',
+ webpage)
+
+ return {
+ 'id': video_id,
+ 'display_id': display_id,
+ 'url': video_url,
+ 'http_headers': {
+ 'Referer': url,
+ },
+ 'title': title,
+ 'thumbnail': thumbnail,
+ 'uploader': uploader,
+ 'upload_date': upload_date,
+ 'duration': duration,
+ 'view_count': view_count,
+ 'comment_count': comment_count,
+ 'categories': categories,
+ 'age_limit': 18,
+ }
from .common import InfoExtractor
from ..utils import (
ExtractorError,
- sanitized_Request,
int_or_none,
+ float_or_none,
)
class WistiaIE(InfoExtractor):
- _VALID_URL = r'https?://(?:fast\.)?wistia\.net/embed/iframe/(?P<id>[a-z0-9]+)'
- _API_URL = 'http://fast.wistia.com/embed/medias/{0:}.json'
+ _VALID_URL = r'(?:wistia:|https?://(?:fast\.)?wistia\.net/embed/iframe/)(?P<id>[a-z0-9]+)'
+ _API_URL = 'http://fast.wistia.com/embed/medias/%s.json'
+ _IFRAME_URL = 'http://fast.wistia.net/embed/iframe/%s'
- _TEST = {
+ _TESTS = [{
'url': 'http://fast.wistia.net/embed/iframe/sh7fpupwlt',
'md5': 'cafeb56ec0c53c18c97405eecb3133df',
'info_dict': {
'timestamp': 1386185018,
'duration': 117,
},
- }
+ }, {
+ 'url': 'wistia:sh7fpupwlt',
+ 'only_matching': True,
+ }, {
+ # with hls video
+ 'url': 'wistia:807fafadvk',
+ 'only_matching': True,
+ }]
def _real_extract(self, url):
video_id = self._match_id(url)
- request = sanitized_Request(self._API_URL.format(video_id))
- request.add_header('Referer', url) # Some videos require this.
- data_json = self._download_json(request, video_id)
+ data_json = self._download_json(
+ self._API_URL % video_id, video_id,
+ # Some videos require this.
+ headers={
+ 'Referer': url if url.startswith('http') else self._IFRAME_URL % video_id,
+ })
+
if data_json.get('error'):
- raise ExtractorError('Error while getting the playlist',
- expected=True)
+ raise ExtractorError(
+ 'Error while getting the playlist', expected=True)
+
data = data_json['media']
title = data['name']
formats = []
thumbnails = []
for a in data['assets']:
+ aurl = a.get('url')
+ if not aurl:
+ continue
astatus = a.get('status')
atype = a.get('type')
- if (astatus is not None and astatus != 2) or atype == 'preview':
+ if (astatus is not None and astatus != 2) or atype in ('preview', 'storyboard'):
continue
elif atype in ('still', 'still_image'):
thumbnails.append({
- 'url': a['url'],
- 'resolution': '%dx%d' % (a['width'], a['height']),
+ 'url': aurl,
+ 'width': int_or_none(a.get('width')),
+ 'height': int_or_none(a.get('height')),
})
else:
+ aext = a.get('ext')
+ is_m3u8 = a.get('container') == 'm3u8' or aext == 'm3u8'
formats.append({
'format_id': atype,
- 'url': a['url'],
+ 'url': aurl,
'tbr': int_or_none(a.get('bitrate')),
'vbr': int_or_none(a.get('opt_vbitrate')),
'width': int_or_none(a.get('width')),
'filesize': int_or_none(a.get('size')),
'vcodec': a.get('codec'),
'container': a.get('container'),
- 'ext': a.get('ext'),
+ 'ext': 'mp4' if is_m3u8 else aext,
+ 'protocol': 'm3u8' if is_m3u8 else None,
'preference': 1 if atype == 'original' else None,
})
'description': data.get('seoDescription'),
'formats': formats,
'thumbnails': thumbnails,
- 'duration': int_or_none(data.get('duration')),
+ 'duration': float_or_none(data.get('duration')),
'timestamp': int_or_none(data.get('createdAt')),
}
from .common import InfoExtractor
from ..utils import (
int_or_none,
+ float_or_none,
unified_strdate,
)
class WSJIE(InfoExtractor):
- _VALID_URL = r'https?://video-api\.wsj\.com/api-video/player/iframe\.html\?guid=(?P<id>[a-zA-Z0-9-]+)'
+ _VALID_URL = r'''(?x)https?://
+ (?:
+ video-api\.wsj\.com/api-video/player/iframe\.html\?guid=|
+ (?:www\.)?wsj\.com/video/[^/]+/
+ )
+ (?P<id>[a-zA-Z0-9-]+)'''
IE_DESC = 'Wall Street Journal'
- _TEST = {
+ _TESTS = [{
'url': 'http://video-api.wsj.com/api-video/player/iframe.html?guid=1BD01A4C-BFE8-40A5-A42F-8A8AF9898B1A',
- 'md5': '9747d7a6ebc2f4df64b981e1dde9efa9',
+ 'md5': 'e230a5bb249075e40793b655a54a02e4',
'info_dict': {
'id': '1BD01A4C-BFE8-40A5-A42F-8A8AF9898B1A',
'ext': 'mp4',
'duration': 90,
'title': 'Bills Coach Rex Ryan Updates His Old Jets Tattoo',
},
- }
+ }, {
+ 'url': 'http://www.wsj.com/video/can-alphabet-build-a-smarter-city/359DDAA8-9AC1-489C-82E6-0429C1E430E0.html',
+ 'only_matching': True,
+ }]
def _real_extract(self, url):
video_id = self._match_id(url)
- bitrates = [128, 174, 264, 320, 464, 664, 1264]
api_url = (
'http://video-api.wsj.com/api-video/find_all_videos.asp?'
- 'type=guid&count=1&query=%s&'
- 'fields=hls,adZone,thumbnailList,guid,state,secondsUntilStartTime,'
- 'author,description,name,linkURL,videoStillURL,duration,videoURL,'
- 'adCategory,catastrophic,linkShortURL,doctypeID,youtubeID,'
- 'titletag,rssURL,wsj-section,wsj-subsection,allthingsd-section,'
- 'allthingsd-subsection,sm-section,sm-subsection,provider,'
- 'formattedCreationDate,keywords,keywordsOmniture,column,editor,'
- 'emailURL,emailPartnerID,showName,omnitureProgramName,'
- 'omnitureVideoFormat,linkRelativeURL,touchCastID,'
- 'omniturePublishDate,%s') % (
- video_id, ','.join('video%dkMP4Url' % br for br in bitrates))
+ 'type=guid&count=1&query=%s&fields=type,hls,videoMP4List,'
+ 'thumbnailList,author,description,name,duration,videoURL,'
+ 'titletag,formattedCreationDate,keywords,editor' % video_id)
info = self._download_json(api_url, video_id)['items'][0]
-
- # Thumbnails are conveniently in the correct format already
- thumbnails = info.get('thumbnailList')
- creator = info.get('author')
- uploader_id = info.get('editor')
- categories = info.get('keywords')
- duration = int_or_none(info.get('duration'))
- upload_date = unified_strdate(
- info.get('formattedCreationDate'), day_first=False)
title = info.get('name', info.get('titletag'))
- formats = [{
- 'format_id': 'f4m',
- 'format_note': 'f4m (meta URL)',
- 'url': info['videoURL'],
- }]
- if info.get('hls'):
+ formats = []
+
+ f4m_url = info.get('videoURL')
+ if f4m_url:
+ formats.extend(self._extract_f4m_formats(
+ f4m_url, video_id, f4m_id='hds', fatal=False))
+
+ m3u8_url = info.get('hls')
+ if m3u8_url:
formats.extend(self._extract_m3u8_formats(
info['hls'], video_id, ext='mp4',
- preference=0, entry_protocol='m3u8_native'))
- for br in bitrates:
- field = 'video%dkMP4Url' % br
- if info.get(field):
- formats.append({
- 'format_id': 'mp4-%d' % br,
- 'container': 'mp4',
- 'tbr': br,
- 'url': info[field],
- })
+ entry_protocol='m3u8_native', m3u8_id='hls', fatal=False))
+
+ for v in info.get('videoMP4List', []):
+ mp4_url = v.get('url')
+ if not mp4_url:
+ continue
+ tbr = int_or_none(v.get('bitrate'))
+ formats.append({
+ 'url': mp4_url,
+ 'format_id': 'http' + ('-%d' % tbr if tbr else ''),
+ 'tbr': tbr,
+ 'width': int_or_none(v.get('width')),
+ 'height': int_or_none(v.get('height')),
+ 'fps': float_or_none(v.get('fps')),
+ })
self._sort_formats(formats)
return {
'id': video_id,
'formats': formats,
- 'thumbnails': thumbnails,
- 'creator': creator,
- 'uploader_id': uploader_id,
- 'duration': duration,
- 'upload_date': upload_date,
+ # Thumbnails are conveniently in the correct format already
+ 'thumbnails': info.get('thumbnailList'),
+ 'creator': info.get('author'),
+ 'uploader_id': info.get('editor'),
+ 'duration': int_or_none(info.get('duration')),
+ 'upload_date': unified_strdate(info.get(
+ 'formattedCreationDate'), day_first=False),
'title': title,
- 'categories': categories,
+ 'categories': info.get('keywords'),
}
class XFileShareIE(InfoExtractor):
- IE_DESC = 'XFileShare based sites: GorillaVid.in, daclips.in, movpod.in, fastvideo.in, realvid.net, filehoot.com and vidto.me'
- _VALID_URL = r'''(?x)
- https?://(?P<host>(?:www\.)?
- (?:daclips\.in|gorillavid\.in|movpod\.in|fastvideo\.in|realvid\.net|filehoot\.com|vidto\.me|powerwatch\.pw))/
- (?:embed-)?(?P<id>[0-9a-zA-Z]+)(?:-[0-9]+x[0-9]+\.html)?
- '''
+ _SITES = (
+ ('daclips.in', 'DaClips'),
+ ('filehoot.com', 'FileHoot'),
+ ('gorillavid.in', 'GorillaVid'),
+ ('movpod.in', 'MovPod'),
+ ('powerwatch.pw', 'PowerWatch'),
+ ('rapidvideo.ws', 'Rapidvideo.ws'),
+ ('thevideobee.to', 'TheVideoBee'),
+ ('vidto.me', 'Vidto'),
+ ('streamin.to', 'Streamin.To'),
+ )
+
+ IE_DESC = 'XFileShare based sites: %s' % ', '.join(list(zip(*_SITES))[1])
+ _VALID_URL = (r'https?://(?P<host>(?:www\.)?(?:%s))/(?:embed-)?(?P<id>[0-9a-zA-Z]+)'
+ % '|'.join(re.escape(site) for site in list(zip(*_SITES))[0]))
_FILE_NOT_FOUND_REGEX = r'>(?:404 - )?File Not Found<'
'title': 'Micro Pig piglets ready on 16th July 2009-bG0PdrCdxUc',
'thumbnail': 're:http://.*\.jpg',
}
- }, {
- # video with countdown timeout
- 'url': 'http://fastvideo.in/1qmdn1lmsmbw',
- 'md5': '8b87ec3f6564a3108a0e8e66594842ba',
- 'info_dict': {
- 'id': '1qmdn1lmsmbw',
- 'ext': 'mp4',
- 'title': 'Man of Steel - Trailer',
- 'thumbnail': 're:http://.*\.jpg',
- },
- }, {
- 'url': 'http://realvid.net/ctn2y6p2eviw',
- 'md5': 'b2166d2cf192efd6b6d764c18fd3710e',
- 'info_dict': {
- 'id': 'ctn2y6p2eviw',
- 'ext': 'flv',
- 'title': 'rdx 1955',
- 'thumbnail': 're:http://.*\.jpg',
- },
}, {
'url': 'http://movpod.in/0wguyyxi1yca',
'only_matching': True,
class XHamsterIE(InfoExtractor):
- _VALID_URL = r'(?P<proto>https?)://(?:.+?\.)?xhamster\.com/movies/(?P<id>[0-9]+)/(?P<seo>.+?)\.html(?:\?.*)?'
- _TESTS = [
- {
- 'url': 'http://xhamster.com/movies/1509445/femaleagent_shy_beauty_takes_the_bait.html',
- 'info_dict': {
- 'id': '1509445',
- 'ext': 'mp4',
- 'title': 'FemaleAgent Shy beauty takes the bait',
- 'upload_date': '20121014',
- 'uploader': 'Ruseful2011',
- 'duration': 893.52,
- 'age_limit': 18,
- }
+ _VALID_URL = r'(?P<proto>https?)://(?:.+?\.)?xhamster\.com/movies/(?P<id>[0-9]+)/(?P<seo>.*?)\.html(?:\?.*)?'
+ _TESTS = [{
+ 'url': 'http://xhamster.com/movies/1509445/femaleagent_shy_beauty_takes_the_bait.html',
+ 'md5': '8281348b8d3c53d39fffb377d24eac4e',
+ 'info_dict': {
+ 'id': '1509445',
+ 'ext': 'mp4',
+ 'title': 'FemaleAgent Shy beauty takes the bait',
+ 'upload_date': '20121014',
+ 'uploader': 'Ruseful2011',
+ 'duration': 893.52,
+ 'age_limit': 18,
},
- {
- 'url': 'http://xhamster.com/movies/2221348/britney_spears_sexy_booty.html?hd',
- 'info_dict': {
- 'id': '2221348',
- 'ext': 'mp4',
- 'title': 'Britney Spears Sexy Booty',
- 'upload_date': '20130914',
- 'uploader': 'jojo747400',
- 'duration': 200.48,
- 'age_limit': 18,
- }
+ }, {
+ 'url': 'http://xhamster.com/movies/2221348/britney_spears_sexy_booty.html?hd',
+ 'info_dict': {
+ 'id': '2221348',
+ 'ext': 'mp4',
+ 'title': 'Britney Spears Sexy Booty',
+ 'upload_date': '20130914',
+ 'uploader': 'jojo747400',
+ 'duration': 200.48,
+ 'age_limit': 18,
+ },
+ 'params': {
+ 'skip_download': True,
+ },
+ }, {
+ # empty seo
+ 'url': 'http://xhamster.com/movies/5667973/.html',
+ 'info_dict': {
+ 'id': '5667973',
+ 'ext': 'mp4',
+ 'title': '....',
+ 'upload_date': '20160208',
+ 'uploader': 'parejafree',
+ 'duration': 72.0,
+ 'age_limit': 18,
},
- {
- 'url': 'https://xhamster.com/movies/2272726/amber_slayed_by_the_knight.html',
- 'only_matching': True,
+ 'params': {
+ 'skip_download': True,
},
- ]
+ }, {
+ 'url': 'https://xhamster.com/movies/2272726/amber_slayed_by_the_knight.html',
+ 'only_matching': True,
+ }]
def _real_extract(self, url):
def extract_video_url(webpage, name):
webpage = self._download_webpage(url, video_id)
video_url = self._search_regex(
- r'href="(https?://xhamster\.com/movies/%s/[^"]+\.html[^"]*)"' % video_id,
+ r'href="(https?://xhamster\.com/movies/%s/[^"]*\.html[^"]*)"' % video_id,
webpage, 'xhamster url', default=None)
if not video_url:
--- /dev/null
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..compat import compat_urllib_parse_unquote
+from ..utils import int_or_none
+
+
+class XiamiBaseIE(InfoExtractor):
+ _API_BASE_URL = 'http://www.xiami.com/song/playlist/cat/json/id'
+
+ def _download_webpage(self, *args, **kwargs):
+ webpage = super(XiamiBaseIE, self)._download_webpage(*args, **kwargs)
+ if '>Xiami is currently not available in your country.<' in webpage:
+ self.raise_geo_restricted('Xiami is currently not available in your country')
+
+ def _extract_track(self, track, track_id=None):
+ title = track['title']
+ track_url = self._decrypt(track['location'])
+
+ subtitles = {}
+ lyrics_url = track.get('lyric_url') or track.get('lyric')
+ if lyrics_url and lyrics_url.startswith('http'):
+ subtitles['origin'] = [{'url': lyrics_url}]
+
+ return {
+ 'id': track.get('song_id') or track_id,
+ 'url': track_url,
+ 'title': title,
+ 'thumbnail': track.get('pic') or track.get('album_pic'),
+ 'duration': int_or_none(track.get('length')),
+ 'creator': track.get('artist', '').split(';')[0],
+ 'track': title,
+ 'album': track.get('album_name'),
+ 'artist': track.get('artist'),
+ 'subtitles': subtitles,
+ }
+
+ def _extract_tracks(self, item_id, typ=None):
+ playlist = self._download_json(
+ '%s/%s%s' % (self._API_BASE_URL, item_id, '/type/%s' % typ if typ else ''), item_id)
+ return [
+ self._extract_track(track, item_id)
+ for track in playlist['data']['trackList']]
+
+ @staticmethod
+ def _decrypt(origin):
+ n = int(origin[0])
+ origin = origin[1:]
+ short_lenth = len(origin) // n
+ long_num = len(origin) - short_lenth * n
+ l = tuple()
+ for i in range(0, n):
+ length = short_lenth
+ if i < long_num:
+ length += 1
+ l += (origin[0:length], )
+ origin = origin[length:]
+ ans = ''
+ for i in range(0, short_lenth + 1):
+ for j in range(0, n):
+ if len(l[j]) > i:
+ ans += l[j][i]
+ return compat_urllib_parse_unquote(ans).replace('^', '0')
+
+
+class XiamiSongIE(XiamiBaseIE):
+ IE_NAME = 'xiami:song'
+ IE_DESC = '虾米音乐'
+ _VALID_URL = r'https?://(?:www\.)?xiami\.com/song/(?P<id>[0-9]+)'
+ _TESTS = [{
+ 'url': 'http://www.xiami.com/song/1775610518',
+ 'md5': '521dd6bea40fd5c9c69f913c232cb57e',
+ 'info_dict': {
+ 'id': '1775610518',
+ 'ext': 'mp3',
+ 'title': 'Woman',
+ 'thumbnail': r're:http://img\.xiami\.net/images/album/.*\.jpg',
+ 'duration': 265,
+ 'creator': 'HONNE',
+ 'track': 'Woman',
+ 'album': 'Woman',
+ 'artist': 'HONNE',
+ 'subtitles': {
+ 'origin': [{
+ 'ext': 'lrc',
+ }],
+ },
+ },
+ 'skip': 'Georestricted',
+ }, {
+ 'url': 'http://www.xiami.com/song/1775256504',
+ 'md5': '932a3abd45c6aa2b1fdbe028fcb4c4fc',
+ 'info_dict': {
+ 'id': '1775256504',
+ 'ext': 'mp3',
+ 'title': '悟空',
+ 'thumbnail': r're:http://img\.xiami\.net/images/album/.*\.jpg',
+ 'duration': 200,
+ 'creator': '戴荃',
+ 'track': '悟空',
+ 'album': '悟空',
+ 'artist': '戴荃',
+ 'subtitles': {
+ 'origin': [{
+ 'ext': 'lrc',
+ }],
+ },
+ },
+ 'skip': 'Georestricted',
+ }]
+
+ def _real_extract(self, url):
+ return self._extract_tracks(self._match_id(url))[0]
+
+
+class XiamiPlaylistBaseIE(XiamiBaseIE):
+ def _real_extract(self, url):
+ item_id = self._match_id(url)
+ return self.playlist_result(self._extract_tracks(item_id, self._TYPE), item_id)
+
+
+class XiamiAlbumIE(XiamiPlaylistBaseIE):
+ IE_NAME = 'xiami:album'
+ IE_DESC = '虾米音乐 - 专辑'
+ _VALID_URL = r'https?://(?:www\.)?xiami\.com/album/(?P<id>[0-9]+)'
+ _TYPE = '1'
+ _TESTS = [{
+ 'url': 'http://www.xiami.com/album/2100300444',
+ 'info_dict': {
+ 'id': '2100300444',
+ },
+ 'playlist_count': 10,
+ 'skip': 'Georestricted',
+ }, {
+ 'url': 'http://www.xiami.com/album/512288?spm=a1z1s.6843761.1110925389.6.hhE9p9',
+ 'only_matching': True,
+ }]
+
+
+class XiamiArtistIE(XiamiPlaylistBaseIE):
+ IE_NAME = 'xiami:artist'
+ IE_DESC = '虾米音乐 - 歌手'
+ _VALID_URL = r'https?://(?:www\.)?xiami\.com/artist/(?P<id>[0-9]+)'
+ _TYPE = '2'
+ _TEST = {
+ 'url': 'http://www.xiami.com/artist/2132?spm=0.0.0.0.dKaScp',
+ 'info_dict': {
+ 'id': '2132',
+ },
+ 'playlist_count': 20,
+ 'skip': 'Georestricted',
+ }
+
+
+class XiamiCollectionIE(XiamiPlaylistBaseIE):
+ IE_NAME = 'xiami:collection'
+ IE_DESC = '虾米音乐 - 精选集'
+ _VALID_URL = r'https?://(?:www\.)?xiami\.com/collect/(?P<id>[0-9]+)'
+ _TYPE = '3'
+ _TEST = {
+ 'url': 'http://www.xiami.com/collect/156527391?spm=a1z1s.2943601.6856193.12.4jpBnr',
+ 'info_dict': {
+ 'id': '156527391',
+ },
+ 'playlist_mincount': 29,
+ 'skip': 'Georestricted',
+ }
from __future__ import unicode_literals
import re
+import time
from .common import InfoExtractor
from ..compat import (
- compat_chr,
compat_ord,
)
from ..utils import (
int_or_none,
- parse_filesize,
+ parse_duration,
)
'info_dict': {
'id': '4542',
'ext': 'mp3',
- 'title': 'Леонид Агутин-Песенка шофера',
+ 'title': 'Леонид Агутин-Песенка шофёра',
'duration': 156,
'tbr': 320,
'filesize_approx': 5900000,
webpage = self._download_webpage(url, video_id)
artist = self._html_search_regex(
- r'minus_track\.artist="(.+?)"', webpage, 'artist')
+ r'<a[^>]+href="/artist/\d+">([^<]+)</a>', webpage, 'artist')
title = artist + '-' + self._html_search_regex(
- r'minus_track\.title="(.+?)"', webpage, 'title')
- duration = int_or_none(self._html_search_regex(
- r'minus_track\.dur_sec=\'([0-9]*?)\'',
+ r'<span[^>]+class="minustrack-full-title(?:\s+[^"]+)?"[^>]*>([^<]+)', webpage, 'title')
+ duration = parse_duration(self._html_search_regex(
+ r'<span[^>]+class="player-duration(?:\s+[^"]+)?"[^>]*>([^<]+)',
webpage, 'duration', fatal=False))
- filesize_approx = parse_filesize(self._html_search_regex(
- r'<div id="finfo"[^>]*>\s*↓\s*([0-9.]+\s*[a-zA-Z][bB])',
- webpage, 'approximate filesize', fatal=False))
- tbr = int_or_none(self._html_search_regex(
- r'<div class="quality[^"]*"></div>\s*([0-9]+)\s*kbps',
- webpage, 'bitrate', fatal=False))
+ mobj = re.search(
+ r'<div[^>]+class="dw-info(?:\s+[^"]+)?"[^>]*>(?P<tbr>\d+)\s*кбит/c\s+(?P<filesize>[0-9.]+)\s*мб</div>',
+ webpage)
+ tbr = filesize_approx = None
+ if mobj:
+ filesize_approx = float(mobj.group('filesize')) * 1000000
+ tbr = float(mobj.group('tbr'))
view_count = int_or_none(self._html_search_regex(
- r'<div class="quality.*?► ([0-9]+)',
+ r'<span><[^>]+class="icon-chart-bar".*?>(\d+)</span>',
webpage, 'view count', fatal=False))
description = self._html_search_regex(
- r'(?s)<div id="song_texts">(.*?)</div><br',
+ r'(?s)<pre[^>]+id="lyrics-original"[^>]*>(.*?)</pre>',
webpage, 'song lyrics', fatal=False)
if description:
description = re.sub(' *\r *', '\n', description)
- enc_token = self._html_search_regex(
- r'minus_track\.s?tkn="(.+?)"', webpage, 'enc_token')
- token = ''.join(
- c if pos == 3 else compat_chr(compat_ord(c) - 1)
- for pos, c in enumerate(reversed(enc_token)))
- video_url = 'http://x-minus.org/dwlf/%s/%s.mp3' % (video_id, token)
+ k = self._search_regex(
+ r'<div[^>]+id="player-bottom"[^>]+data-k="([^"]+)">', webpage,
+ 'encoded data')
+ h = time.time() / 3600
+ a = sum(map(int, [compat_ord(c) for c in k])) + int(video_id) + h
+ video_url = 'http://x-minus.me/dl/minus?id=%s&tkn2=%df%d' % (video_id, a, h)
return {
'id': video_id,
'title': title,
'url': video_url,
+ # The extension is unknown until actual downloading
+ 'ext': 'mp3',
'duration': duration,
'filesize_approx': filesize_approx,
'tbr': tbr,
clean_html,
ExtractorError,
determine_ext,
- sanitized_Request,
)
}
}
- _ANDROID_USER_AGENT = 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19'
-
def _real_extract(self, url):
video_id = self._match_id(url)
webpage = self._download_webpage(url, video_id)
if mobj:
raise ExtractorError('%s said: %s' % (self.IE_NAME, clean_html(mobj.group(1))), expected=True)
- video_url = compat_urllib_parse_unquote(
- self._search_regex(r'flv_url=(.+?)&', webpage, 'video URL'))
video_title = self._html_search_regex(
r'<title>(.*?)\s+-\s+XVID', webpage, 'title')
video_thumbnail = self._search_regex(
r'url_bigthumb=(.+?)&', webpage, 'thumbnail', fatal=False)
- formats = [{
- 'url': video_url,
- }]
+ formats = []
- android_req = sanitized_Request(url)
- android_req.add_header('User-Agent', self._ANDROID_USER_AGENT)
- android_webpage = self._download_webpage(android_req, video_id, fatal=False)
+ video_url = compat_urllib_parse_unquote(self._search_regex(
+ r'flv_url=(.+?)&', webpage, 'video URL', default=''))
+ if video_url:
+ formats.append({'url': video_url})
- if android_webpage is not None:
- player_params_str = self._search_regex(
- 'mobileReplacePlayerDivTwoQual\(([^)]+)\)',
- android_webpage, 'player parameters', default='')
- player_params = list(map(lambda s: s.strip(' \''), player_params_str.split(',')))
- if player_params:
- formats.extend([{
- 'url': param,
- 'preference': -10,
- } for param in player_params if determine_ext(param) == 'mp4'])
+ player_args = self._search_regex(
+ r'(?s)new\s+HTML5Player\((.+?)\)', webpage, ' html5 player', default=None)
+ if player_args:
+ for arg in player_args.split(','):
+ format_url = self._search_regex(
+ r'(["\'])(?P<url>https?://.+?)\1', arg, 'url',
+ default=None, group='url')
+ if not format_url:
+ continue
+ ext = determine_ext(format_url)
+ if ext == 'mp4':
+ formats.append({'url': format_url})
+ elif ext == 'm3u8':
+ formats.extend(self._extract_m3u8_formats(
+ format_url, video_id, 'mp4',
+ entry_protocol='m3u8_native', m3u8_id='hls', fatal=False))
self._sort_formats(formats)
'id': video_id,
'formats': formats,
'title': video_title,
- 'ext': 'flv',
'thumbnail': video_thumbnail,
'age_limit': 18,
}
class YahooIE(InfoExtractor):
IE_DESC = 'Yahoo screen and movies'
- _VALID_URL = r'(?P<url>(?P<host>https?://(?:[a-zA-Z]{2}\.)?[\da-zA-Z_-]+\.yahoo\.com)/(?:[^/]+/)*(?P<display_id>.+)?-(?P<id>[0-9]+)(?:-[a-z]+)?\.html)'
+ _VALID_URL = r'(?P<url>(?P<host>https?://(?:[a-zA-Z]{2}\.)?[\da-zA-Z_-]+\.yahoo\.com)/(?:[^/]+/)*(?P<display_id>.+)?-(?P<id>[0-9]+)(?:-[a-z]+)?(?:\.html)?)'
_TESTS = [
{
'url': 'http://screen.yahoo.com/julian-smith-travis-legg-watch-214727115.html',
},
{
'url': 'http://screen.yahoo.com/wired/codefellas-s1-ep12-cougar-lies-103000935.html',
- 'md5': 'd6e6fc6e1313c608f316ddad7b82b306',
+ 'md5': 'c3466d2b6d5dd6b9f41ba9ed04c24b23',
'info_dict': {
'id': 'd1dedf8c-d58c-38c3-8963-e899929ae0a9',
'ext': 'mp4',
},
{
'url': 'https://screen.yahoo.com/community/community-sizzle-reel-203225340.html?format=embed',
- 'md5': '60e8ac193d8fb71997caa8fce54c6460',
+ 'md5': '75ffabdb87c16d4ffe8c036dc4d1c136',
'info_dict': {
'id': '4fe78544-8d48-39d8-97cd-13f205d9fcdb',
'ext': 'mp4',
}
},
{
- 'url': 'https://tw.screen.yahoo.com/election-2014-askmayor/敢問市長-黃秀霜批賴清德-非常高傲-033009720.html',
- 'md5': '3a09cf59349cfaddae1797acc3c087fc',
+ 'url': 'https://tw.news.yahoo.com/%E6%95%A2%E5%95%8F%E5%B8%82%E9%95%B7%20%E9%BB%83%E7%A7%80%E9%9C%9C%E6%89%B9%E8%B3%B4%E6%B8%85%E5%BE%B7%20%E9%9D%9E%E5%B8%B8%E9%AB%98%E5%82%B2-034024051.html',
+ 'md5': '9035d38f88b1782682a3e89f985be5bb',
'info_dict': {
'id': 'cac903b3-fcf4-3c14-b632-643ab541712f',
'ext': 'mp4',
'title': '敢問市長／黃秀霜批賴清德「非常高傲」',
'description': '直言台南沒捷運 交通居五都之末',
'duration': 396,
- }
+ },
},
{
'url': 'https://uk.screen.yahoo.com/editor-picks/cute-raccoon-freed-drain-using-091756545.html',
'title': 'Program that makes hockey more affordable not offered in Manitoba',
'description': 'md5:c54a609f4c078d92b74ffb9bf1f496f4',
'duration': 121,
- }
+ },
+ 'skip': 'Video gone',
}, {
'url': 'https://ca.finance.yahoo.com/news/hackers-sony-more-trouble-well-154609075.html',
- 'md5': '226a895aae7e21b0129e2a2006fe9690',
'info_dict': {
- 'id': 'e624c4bc-3389-34de-9dfc-025f74943409',
- 'ext': 'mp4',
- 'title': '\'The Interview\' TV Spot: War',
- 'description': 'The Interview',
- 'duration': 30,
- }
+ 'id': '154609075',
+ },
+ 'playlist': [{
+ 'md5': 'f8e336c6b66f503282e5f719641d6565',
+ 'info_dict': {
+ 'id': 'e624c4bc-3389-34de-9dfc-025f74943409',
+ 'ext': 'mp4',
+ 'title': '\'The Interview\' TV Spot: War',
+ 'description': 'The Interview',
+ 'duration': 30,
+ },
+ }, {
+ 'md5': '958bcb90b4d6df71c56312137ee1cd5a',
+ 'info_dict': {
+ 'id': '1fc8ada0-718e-3abe-a450-bf31f246d1a9',
+ 'ext': 'mp4',
+ 'title': '\'The Interview\' TV Spot: Guys',
+ 'description': 'The Interview',
+ 'duration': 30,
+ },
+ }],
}, {
'url': 'http://news.yahoo.com/video/china-moses-crazy-blues-104538833.html',
'md5': '88e209b417f173d86186bef6e4d1f160',
'title': 'Connect the Dots: Dark Side of Virgo',
'description': 'md5:1428185051cfd1949807ad4ff6d3686a',
'duration': 201,
- }
+ },
+ 'skip': 'Domain name in.lifestyle.yahoo.com gone',
}, {
'url': 'https://www.yahoo.com/movies/v/true-story-trailer-173000497.html',
- 'md5': '989396ae73d20c6f057746fb226aa215',
+ 'md5': 'b17ac378b1134fa44370fb27db09a744',
'info_dict': {
'id': '071c4013-ce30-3a93-a5b2-e0413cd4a9d1',
'ext': 'mp4',
'ext': 'flv',
'description': 'md5:df390f70a9ba7c95ff1daace988f0d8d',
'title': 'Tyler Kalinoski hits buzzer-beater to lift Davidson',
+ 'upload_date': '20150313',
+ 'uploader': 'NBCU-SPORTS',
+ 'timestamp': 1426270238,
}
}, {
'url': 'https://tw.news.yahoo.com/-100120367.html',
}, {
# Query result is embedded in webpage, but explicit request to video API fails with geo restriction
'url': 'https://screen.yahoo.com/community/communitary-community-episode-1-ladders-154501237.html',
- 'md5': '4fbafb9c9b6f07aa8f870629f6671b35',
+ 'md5': '1ddbf7c850777548438e5c4f147c7b8c',
'info_dict': {
'id': '1f32853c-a271-3eef-8cb6-f6d6872cb504',
'ext': 'mp4',
'description': 'While they play feuding fathers in \'Daddy\'s Home,\' star Will Ferrell & Mark Wahlberg share their true feelings on parenthood.',
},
},
+ {
+ # config['models']['applet_model']['data']['sapi'] has no query
+ 'url': 'https://www.yahoo.com/music/livenation/event/galactic-2016',
+ 'md5': 'dac0c72d502bc5facda80c9e6d5c98db',
+ 'info_dict': {
+ 'id': 'a6015640-e9e5-3efb-bb60-05589a183919',
+ 'ext': 'mp4',
+ 'description': 'Galactic',
+ 'title': 'Dolla Diva (feat. Maggie Koerner)',
+ },
+ },
]
def _real_extract(self, url):
page_id = mobj.group('id')
url = mobj.group('url')
host = mobj.group('host')
- webpage = self._download_webpage(url, display_id)
+ webpage, urlh = self._download_webpage_handle(url, display_id)
+ if 'err=404' in urlh.geturl():
+ raise ExtractorError('Video gone', expected=True)
# Look for iframed media first
- iframe_m = re.search(r'<iframe[^>]+src="(/video/.+?-\d+\.html\?format=embed.*?)"', webpage)
- if iframe_m:
+ entries = []
+ iframe_urls = re.findall(r'<iframe[^>]+src="(/video/.+?-\d+\.html\?format=embed.*?)"', webpage)
+ for idx, iframe_url in enumerate(iframe_urls):
iframepage = self._download_webpage(
- host + iframe_m.group(1), display_id, 'Downloading iframe webpage')
+ host + iframe_url, display_id,
+ note='Downloading iframe webpage for video #%d' % idx)
items_json = self._search_regex(
r'mediaItems: (\[.+?\])$', iframepage, 'items', flags=re.MULTILINE, default=None)
if items_json:
items = json.loads(items_json)
video_id = items[0]['id']
- return self._get_info(video_id, display_id, webpage)
+ entries.append(self._get_info(video_id, display_id, webpage))
+ if entries:
+ return self.playlist_result(entries, page_id)
+
# Look for NBCSports iframes
nbc_sports_url = NBCSportsVPlayerIE._extract_url(webpage)
if nbc_sports_url:
config = self._parse_json(config_json, display_id, fatal=False)
if config:
sapi = config.get('models', {}).get('applet_model', {}).get('data', {}).get('sapi')
- if sapi:
+ if sapi and 'query' in sapi:
return self._extract_info(display_id, sapi, webpage)
items_json = self._search_regex(
ExtractorError,
int_or_none,
float_or_none,
- sanitized_Request,
- urlencode_postdata,
)
class YandexMusicBaseIE(InfoExtractor):
@staticmethod
def _handle_error(response):
- error = response.get('error')
- if error:
- raise ExtractorError(error, expected=True)
+ if isinstance(response, dict):
+ error = response.get('error')
+ if error:
+ raise ExtractorError(error, expected=True)
+ if response.get('type') == 'captcha' or 'captcha' in response:
+ YandexMusicBaseIE._raise_captcha()
+
+ @staticmethod
+ def _raise_captcha():
+ raise ExtractorError(
+ 'YandexMusic has considered youtube-dl requests automated and '
+ 'asks you to solve a CAPTCHA. You can either wait for some '
+ 'time until unblocked and optionally use --sleep-interval '
+ 'in future or alternatively you can go to https://music.yandex.ru/ '
+ 'solve CAPTCHA, then export cookies and pass cookie file to '
+ 'youtube-dl with --cookies',
+ expected=True)
+
+ def _download_webpage(self, *args, **kwargs):
+ webpage = super(YandexMusicBaseIE, self)._download_webpage(*args, **kwargs)
+ if 'Нам очень жаль, но запросы, поступившие с вашего IP-адреса, похожи на автоматические.' in webpage:
+ self._raise_captcha()
+ return webpage
def _download_json(self, *args, **kwargs):
response = super(YandexMusicBaseIE, self)._download_json(*args, **kwargs)
'album_artist': 'Carlo Ambrosio',
'artist': 'Carlo Ambrosio & Fabio Di Bari, Carlo Ambrosio',
'release_year': '2009',
- }
+ },
+ 'skip': 'Travis CI servers blocked by YandexMusic',
}
def _get_track_url(self, storage_dir, track_id):
'title': 'Carlo Ambrosio - Gypsy Soul (2009)',
},
'playlist_count': 50,
+ 'skip': 'Travis CI servers blocked by YandexMusic',
}
def _real_extract(self, url):
class YandexMusicPlaylistIE(YandexMusicPlaylistBaseIE):
IE_NAME = 'yandexmusic:playlist'
IE_DESC = 'Яндекс.Музыка - Плейлист'
- _VALID_URL = r'https?://music\.yandex\.(?:ru|kz|ua|by)/users/[^/]+/playlists/(?P<id>\d+)'
+ _VALID_URL = r'https?://music\.yandex\.(?P<tld>ru|kz|ua|by)/users/(?P<user>[^/]+)/playlists/(?P<id>\d+)'
_TESTS = [{
'url': 'http://music.yandex.ru/users/music.partners/playlists/1245',
'description': 'md5:3b9f27b0efbe53f2ee1e844d07155cc9',
},
'playlist_count': 6,
+ 'skip': 'Travis CI servers blocked by YandexMusic',
}, {
# playlist exceeding the limit of 150 tracks shipped with webpage (see
# https://github.com/rg3/youtube-dl/issues/6666)
'id': '1036',
'title': 'Музыка 90-х',
},
- 'playlist_count': 310,
+ 'playlist_mincount': 300,
+ 'skip': 'Travis CI servers blocked by YandexMusic',
}]
def _real_extract(self, url):
- playlist_id = self._match_id(url)
-
- webpage = self._download_webpage(url, playlist_id)
-
- mu = self._parse_json(
- self._search_regex(
- r'var\s+Mu\s*=\s*({.+?});\s*</script>', webpage, 'player'),
- playlist_id)
-
- playlist = mu['pageData']['playlist']
- tracks, track_ids = playlist['tracks'], playlist['trackIds']
-
- # tracks dictionary shipped with webpage is limited to 150 tracks,
+ mobj = re.match(self._VALID_URL, url)
+ tld = mobj.group('tld')
+ user = mobj.group('user')
+ playlist_id = mobj.group('id')
+
+ playlist = self._download_json(
+ 'https://music.yandex.%s/handlers/playlist.jsx' % tld,
+ playlist_id, 'Downloading missing tracks JSON',
+ fatal=False,
+ headers={
+ 'Referer': url,
+ 'X-Requested-With': 'XMLHttpRequest',
+ 'X-Retpath-Y': url,
+ },
+ query={
+ 'owner': user,
+ 'kinds': playlist_id,
+ 'light': 'true',
+ 'lang': tld,
+ 'external-domain': 'music.yandex.%s' % tld,
+ 'overembed': 'false',
+ })['playlist']
+
+ tracks, track_ids = playlist['tracks'], map(compat_str, playlist['trackIds'])
+
+ # tracks dictionary shipped with playlist.jsx API is limited to 150 tracks,
# missing tracks should be retrieved manually.
if len(tracks) < len(track_ids):
- present_track_ids = set([compat_str(track['id']) for track in tracks if track.get('id')])
- missing_track_ids = set(map(compat_str, track_ids)) - set(present_track_ids)
- request = sanitized_Request(
- 'https://music.yandex.ru/handlers/track-entries.jsx',
- urlencode_postdata({
+ present_track_ids = set([
+ compat_str(track['id'])
+ for track in tracks if track.get('id')])
+ missing_track_ids = [
+ track_id for track_id in track_ids
+ if track_id not in present_track_ids]
+ missing_tracks = self._download_json(
+ 'https://music.yandex.%s/handlers/track-entries.jsx' % tld,
+ playlist_id, 'Downloading missing tracks JSON',
+ fatal=False,
+ headers={
+ 'Referer': url,
+ 'X-Requested-With': 'XMLHttpRequest',
+ },
+ query={
'entries': ','.join(missing_track_ids),
- 'lang': mu.get('settings', {}).get('lang', 'en'),
- 'external-domain': 'music.yandex.ru',
+ 'lang': tld,
+ 'external-domain': 'music.yandex.%s' % tld,
'overembed': 'false',
- 'sign': mu.get('authData', {}).get('user', {}).get('sign'),
'strict': 'true',
- }))
- request.add_header('Referer', url)
- request.add_header('X-Requested-With', 'XMLHttpRequest')
-
- missing_tracks = self._download_json(
- request, playlist_id, 'Downloading missing tracks JSON', fatal=False)
+ })
if missing_tracks:
tracks.extend(missing_tracks)
return self.playlist_result(
self._build_playlist(tracks),
compat_str(playlist_id),
- playlist['title'], playlist.get('description'))
+ playlist.get('title'), playlist.get('description'))
'format_id': self.get_format_name(fm),
'ext': self.parse_ext_l(fm),
'filesize': int(seg['size']),
+ 'width': stream.get('width'),
+ 'height': stream.get('height'),
})
return {
'139': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'abr': 48, 'preference': -50, 'container': 'm4a_dash'},
'140': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'abr': 128, 'preference': -50, 'container': 'm4a_dash'},
'141': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'abr': 256, 'preference': -50, 'container': 'm4a_dash'},
+ '256': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'preference': -50, 'container': 'm4a_dash'},
+ '258': {'ext': 'm4a', 'format_note': 'DASH audio', 'acodec': 'aac', 'preference': -50, 'container': 'm4a_dash'},
# Dash webm
'167': {'ext': 'webm', 'height': 360, 'width': 640, 'format_note': 'DASH video', 'container': 'webm', 'vcodec': 'vp8', 'preference': -40},
if video_description:
video_description = re.sub(r'''(?x)
<a\s+
- (?:[a-zA-Z-]+="[^"]+"\s+)*?
+ (?:[a-zA-Z-]+="[^"]*"\s+)*?
(?:title|href)="([^"]+)"\s+
- (?:[a-zA-Z-]+="[^"]+"\s+)*?
+ (?:[a-zA-Z-]+="[^"]*"\s+)*?
class="(?:yt-uix-redirect-link|yt-uix-sessionlink[^"]*)"[^>]*>
[^<]+\.{3}\s*
</a>
_EXTRA_QUERY_ARGS = {'search_sort': 'video_date_uploaded'}
-class YoutubeSearchURLIE(InfoExtractor):
+class YoutubeSearchURLIE(YoutubePlaylistBaseInfoExtractor):
IE_DESC = 'YouTube.com search URLs'
IE_NAME = 'youtube:search_url'
_VALID_URL = r'https?://(?:www\.)?youtube\.com/results\?(.*?&)?(?:search_query|q)=(?P<query>[^&]+)(?:[&]|$)'
+ _VIDEO_RE = r'href="\s*/watch\?v=(?P<id>[0-9A-Za-z_-]{11})(?:[^"]*"[^>]+\btitle="(?P<title>[^"]+))?'
_TESTS = [{
'url': 'https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video',
'playlist_mincount': 5,
def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
query = compat_urllib_parse_unquote_plus(mobj.group('query'))
-
webpage = self._download_webpage(url, query)
- result_code = self._search_regex(
- r'(?s)<ol[^>]+class="item-section"(.*?)</ol>', webpage, 'result HTML')
-
- part_codes = re.findall(
- r'(?s)<h3[^>]+class="[^"]*yt-lockup-title[^"]*"[^>]*>(.*?)</h3>', result_code)
- entries = []
- for part_code in part_codes:
- part_title = self._html_search_regex(
- [r'(?s)title="([^"]+)"', r'>([^<]+)</a>'], part_code, 'item title', fatal=False)
- part_url_snippet = self._html_search_regex(
- r'(?s)href="([^"]+)"', part_code, 'item URL')
- part_url = compat_urlparse.urljoin(
- 'https://www.youtube.com/', part_url_snippet)
- entries.append({
- '_type': 'url',
- 'url': part_url,
- 'title': part_title,
- })
-
- return {
- '_type': 'playlist',
- 'entries': entries,
- 'title': query,
- }
+ return self.playlist_result(self._process_page(webpage), playlist_title=query)
class YoutubeShowIE(YoutubePlaylistsBaseInfoExtractor):
network.add_option(
'--proxy', dest='proxy',
default=None, metavar='URL',
- help='Use the specified HTTP/HTTPS proxy. Pass in an empty string (--proxy "") for direct connection')
+ help='Use the specified HTTP/HTTPS/SOCKS proxy. To enable experimental '
+ 'SOCKS proxy, specify a proper scheme. For example '
+ 'socks5://127.0.0.1:1080/. Pass in an empty string (--proxy "") '
+ 'for direct connection')
network.add_option(
'--socket-timeout',
dest='socket_timeout', type=float, default=None, metavar='SECONDS',
downloader = optparse.OptionGroup(parser, 'Download Options')
downloader.add_option(
- '-r', '--rate-limit',
- dest='ratelimit', metavar='LIMIT',
+ '-r', '--limit-rate', '--rate-limit',
+ dest='ratelimit', metavar='RATE',
help='Maximum download rate in bytes per second (e.g. 50K or 4.2M)')
downloader.add_option(
'-R', '--retries',
action='store_true', dest='writeannotations', default=False,
help='Write video annotations to a .annotations.xml file')
filesystem.add_option(
- '--load-info',
+ '--load-info-json', '--load-info',
dest='load_info_filename', metavar='FILE',
help='JSON file containing the video information (created with the "--write-info-json" option)')
filesystem.add_option(
import subprocess
from .common import PostProcessor
-from ..compat import shlex_quote
+from ..compat import compat_shlex_quote
from ..utils import PostProcessingError
if '{}' not in cmd:
cmd += ' {}'
- cmd = cmd.replace('{}', shlex_quote(information['filepath']))
+ cmd = cmd.replace('{}', compat_shlex_quote(information['filepath']))
self._downloader.to_screen('[exec] Executing command: %s' % cmd)
retCode = subprocess.call(cmd, shell=True)
class FFmpegMetadataPP(FFmpegPostProcessor):
def run(self, info):
metadata = {}
- if info.get('title') is not None:
- metadata['title'] = info['title']
- if info.get('upload_date') is not None:
- metadata['date'] = info['upload_date']
- if info.get('artist') is not None:
- metadata['artist'] = info['artist']
- elif info.get('uploader') is not None:
- metadata['artist'] = info['uploader']
- elif info.get('uploader_id') is not None:
- metadata['artist'] = info['uploader_id']
- if info.get('description') is not None:
- metadata['description'] = info['description']
- metadata['comment'] = info['description']
- if info.get('webpage_url') is not None:
- metadata['purl'] = info['webpage_url']
- if info.get('album') is not None:
- metadata['album'] = info['album']
+
+ def add(meta_list, info_list=None):
+ if not info_list:
+ info_list = meta_list
+ if not isinstance(meta_list, (list, tuple)):
+ meta_list = (meta_list,)
+ if not isinstance(info_list, (list, tuple)):
+ info_list = (info_list,)
+ for info_f in info_list:
+ if info.get(info_f) is not None:
+ for meta_f in meta_list:
+ metadata[meta_f] = info[info_f]
+ break
+
+ add('title', ('track', 'title'))
+ add('date', 'upload_date')
+ add(('description', 'comment'), 'description')
+ add('purl', 'webpage_url')
+ add('track', 'track_number')
+ add('artist', ('artist', 'creator', 'uploader', 'uploader_id'))
+ add('genre')
+ add('album')
+ add('album_artist')
+ add('disc', 'disc_number')
if not metadata:
self._downloader.to_screen('[ffmpeg] There isn\'t any metadata to add')
--- /dev/null
+# Public Domain SOCKS proxy protocol implementation
+# Adapted from https://gist.github.com/bluec0re/cafd3764412967417fd3
+
+from __future__ import unicode_literals
+
+# References:
+# SOCKS4 protocol http://www.openssh.com/txt/socks4.protocol
+# SOCKS4A protocol http://www.openssh.com/txt/socks4a.protocol
+# SOCKS5 protocol https://tools.ietf.org/html/rfc1928
+# SOCKS5 username/password authentication https://tools.ietf.org/html/rfc1929
+
+import collections
+import socket
+
+from .compat import (
+ compat_ord,
+ compat_struct_pack,
+ compat_struct_unpack,
+)
+
+__author__ = 'Timo Schmid <coding@timoschmid.de>'
+
+SOCKS4_VERSION = 4
+SOCKS4_REPLY_VERSION = 0x00
+# Excerpt from SOCKS4A protocol:
+# if the client cannot resolve the destination host's domain name to find its
+# IP address, it should set the first three bytes of DSTIP to NULL and the last
+# byte to a non-zero value.
+SOCKS4_DEFAULT_DSTIP = compat_struct_pack('!BBBB', 0, 0, 0, 0xFF)
+
+SOCKS5_VERSION = 5
+SOCKS5_USER_AUTH_VERSION = 0x01
+SOCKS5_USER_AUTH_SUCCESS = 0x00
+
+
+class Socks4Command(object):
+ CMD_CONNECT = 0x01
+ CMD_BIND = 0x02
+
+
+class Socks5Command(Socks4Command):
+ CMD_UDP_ASSOCIATE = 0x03
+
+
+class Socks5Auth(object):
+ AUTH_NONE = 0x00
+ AUTH_GSSAPI = 0x01
+ AUTH_USER_PASS = 0x02
+ AUTH_NO_ACCEPTABLE = 0xFF # For server response
+
+
+class Socks5AddressType(object):
+ ATYP_IPV4 = 0x01
+ ATYP_DOMAINNAME = 0x03
+ ATYP_IPV6 = 0x04
+
+
+class ProxyError(IOError):
+ ERR_SUCCESS = 0x00
+
+ def __init__(self, code=None, msg=None):
+ if code is not None and msg is None:
+ msg = self.CODES.get(code) and 'unknown error'
+ super(ProxyError, self).__init__(code, msg)
+
+
+class InvalidVersionError(ProxyError):
+ def __init__(self, expected_version, got_version):
+ msg = ('Invalid response version from server. Expected {0:02x} got '
+ '{1:02x}'.format(expected_version, got_version))
+ super(InvalidVersionError, self).__init__(0, msg)
+
+
+class Socks4Error(ProxyError):
+ ERR_SUCCESS = 90
+
+ CODES = {
+ 91: 'request rejected or failed',
+ 92: 'request rejected becasue SOCKS server cannot connect to identd on the client',
+ 93: 'request rejected because the client program and identd report different user-ids'
+ }
+
+
+class Socks5Error(ProxyError):
+ ERR_GENERAL_FAILURE = 0x01
+
+ CODES = {
+ 0x01: 'general SOCKS server failure',
+ 0x02: 'connection not allowed by ruleset',
+ 0x03: 'Network unreachable',
+ 0x04: 'Host unreachable',
+ 0x05: 'Connection refused',
+ 0x06: 'TTL expired',
+ 0x07: 'Command not supported',
+ 0x08: 'Address type not supported',
+ 0xFE: 'unknown username or invalid password',
+ 0xFF: 'all offered authentication methods were rejected'
+ }
+
+
+class ProxyType(object):
+ SOCKS4 = 0
+ SOCKS4A = 1
+ SOCKS5 = 2
+
+Proxy = collections.namedtuple('Proxy', (
+ 'type', 'host', 'port', 'username', 'password', 'remote_dns'))
+
+
+class sockssocket(socket.socket):
+ def __init__(self, *args, **kwargs):
+ self._proxy = None
+ super(sockssocket, self).__init__(*args, **kwargs)
+
+ def setproxy(self, proxytype, addr, port, rdns=True, username=None, password=None):
+ assert proxytype in (ProxyType.SOCKS4, ProxyType.SOCKS4A, ProxyType.SOCKS5)
+
+ self._proxy = Proxy(proxytype, addr, port, username, password, rdns)
+
+ def recvall(self, cnt):
+ data = b''
+ while len(data) < cnt:
+ cur = self.recv(cnt - len(data))
+ if not cur:
+ raise IOError('{0} bytes missing'.format(cnt - len(data)))
+ data += cur
+ return data
+
+ def _recv_bytes(self, cnt):
+ data = self.recvall(cnt)
+ return compat_struct_unpack('!{0}B'.format(cnt), data)
+
+ @staticmethod
+ def _len_and_data(data):
+ return compat_struct_pack('!B', len(data)) + data
+
+ def _check_response_version(self, expected_version, got_version):
+ if got_version != expected_version:
+ self.close()
+ raise InvalidVersionError(expected_version, got_version)
+
+ def _resolve_address(self, destaddr, default, use_remote_dns):
+ try:
+ return socket.inet_aton(destaddr)
+ except socket.error:
+ if use_remote_dns and self._proxy.remote_dns:
+ return default
+ else:
+ return socket.inet_aton(socket.gethostbyname(destaddr))
+
+ def _setup_socks4(self, address, is_4a=False):
+ destaddr, port = address
+
+ ipaddr = self._resolve_address(destaddr, SOCKS4_DEFAULT_DSTIP, use_remote_dns=is_4a)
+
+ packet = compat_struct_pack('!BBH', SOCKS4_VERSION, Socks4Command.CMD_CONNECT, port) + ipaddr
+
+ username = (self._proxy.username or '').encode('utf-8')
+ packet += username + b'\x00'
+
+ if is_4a and self._proxy.remote_dns:
+ packet += destaddr.encode('utf-8') + b'\x00'
+
+ self.sendall(packet)
+
+ version, resp_code, dstport, dsthost = compat_struct_unpack('!BBHI', self.recvall(8))
+
+ self._check_response_version(SOCKS4_REPLY_VERSION, version)
+
+ if resp_code != Socks4Error.ERR_SUCCESS:
+ self.close()
+ raise Socks4Error(resp_code)
+
+ return (dsthost, dstport)
+
+ def _setup_socks4a(self, address):
+ self._setup_socks4(address, is_4a=True)
+
+ def _socks5_auth(self):
+ packet = compat_struct_pack('!B', SOCKS5_VERSION)
+
+ auth_methods = [Socks5Auth.AUTH_NONE]
+ if self._proxy.username and self._proxy.password:
+ auth_methods.append(Socks5Auth.AUTH_USER_PASS)
+
+ packet += compat_struct_pack('!B', len(auth_methods))
+ packet += compat_struct_pack('!{0}B'.format(len(auth_methods)), *auth_methods)
+
+ self.sendall(packet)
+
+ version, method = self._recv_bytes(2)
+
+ self._check_response_version(SOCKS5_VERSION, version)
+
+ if method == Socks5Auth.AUTH_NO_ACCEPTABLE:
+ self.close()
+ raise Socks5Error(method)
+
+ if method == Socks5Auth.AUTH_USER_PASS:
+ username = self._proxy.username.encode('utf-8')
+ password = self._proxy.password.encode('utf-8')
+ packet = compat_struct_pack('!B', SOCKS5_USER_AUTH_VERSION)
+ packet += self._len_and_data(username) + self._len_and_data(password)
+ self.sendall(packet)
+
+ version, status = self._recv_bytes(2)
+
+ self._check_response_version(SOCKS5_USER_AUTH_VERSION, version)
+
+ if status != SOCKS5_USER_AUTH_SUCCESS:
+ self.close()
+ raise Socks5Error(Socks5Error.ERR_GENERAL_FAILURE)
+
+ def _setup_socks5(self, address):
+ destaddr, port = address
+
+ ipaddr = self._resolve_address(destaddr, None, use_remote_dns=True)
+
+ self._socks5_auth()
+
+ reserved = 0
+ packet = compat_struct_pack('!BBB', SOCKS5_VERSION, Socks5Command.CMD_CONNECT, reserved)
+ if ipaddr is None:
+ destaddr = destaddr.encode('utf-8')
+ packet += compat_struct_pack('!B', Socks5AddressType.ATYP_DOMAINNAME)
+ packet += self._len_and_data(destaddr)
+ else:
+ packet += compat_struct_pack('!B', Socks5AddressType.ATYP_IPV4) + ipaddr
+ packet += compat_struct_pack('!H', port)
+
+ self.sendall(packet)
+
+ version, status, reserved, atype = self._recv_bytes(4)
+
+ self._check_response_version(SOCKS5_VERSION, version)
+
+ if status != Socks5Error.ERR_SUCCESS:
+ self.close()
+ raise Socks5Error(status)
+
+ if atype == Socks5AddressType.ATYP_IPV4:
+ destaddr = self.recvall(4)
+ elif atype == Socks5AddressType.ATYP_DOMAINNAME:
+ alen = compat_ord(self.recv(1))
+ destaddr = self.recvall(alen)
+ elif atype == Socks5AddressType.ATYP_IPV6:
+ destaddr = self.recvall(16)
+ destport = compat_struct_unpack('!H', self.recvall(2))[0]
+
+ return (destaddr, destport)
+
+ def _make_proxy(self, connect_func, address):
+ if not self._proxy:
+ return connect_func(self, address)
+
+ result = connect_func(self, (self._proxy.host, self._proxy.port))
+ if result != 0 and result is not None:
+ return result
+ setup_funcs = {
+ ProxyType.SOCKS4: self._setup_socks4,
+ ProxyType.SOCKS4A: self._setup_socks4a,
+ ProxyType.SOCKS5: self._setup_socks5,
+ }
+ setup_funcs[self._proxy.type](address)
+ return result
+
+ def connect(self, address):
+ self._make_proxy(socket.socket.connect, address)
+
+ def connect_ex(self, address):
+ return self._make_proxy(socket.socket.connect_ex, address)
import io
import zlib
-from .compat import compat_str
+from .compat import (
+ compat_str,
+ compat_struct_unpack,
+)
from .utils import (
ExtractorError,
- struct_unpack,
)
file_contents[:1])
# Determine number of bits in framesize rectangle
- framesize_nbits = struct_unpack('!B', content[:1])[0] >> 3
+ framesize_nbits = compat_struct_unpack('!B', content[:1])[0] >> 3
framesize_len = (5 + 4 * framesize_nbits + 7) // 8
pos = framesize_len + 2 + 2
while pos < len(content):
- header16 = struct_unpack('<H', content[pos:pos + 2])[0]
+ header16 = compat_struct_unpack('<H', content[pos:pos + 2])[0]
pos += 2
tag_code = header16 >> 6
tag_len = header16 & 0x3f
if tag_len == 0x3f:
- tag_len = struct_unpack('<I', content[pos:pos + 4])[0]
+ tag_len = compat_struct_unpack('<I', content[pos:pos + 4])[0]
pos += 4
assert pos + tag_len <= len(content), \
('Tag %d ends at %d+%d - that\'s longer than the file (%d)'
for _ in range(5):
buf = reader.read(1)
assert len(buf) == 1
- b = struct_unpack('<B', buf)[0]
+ b = compat_struct_unpack('<B', buf)[0]
res = res | ((b & 0x7f) << shift)
if b & 0x80 == 0:
break
bs = reader.read(3)
assert len(bs) == 3
last_byte = b'\xff' if (ord(bs[2:3]) >= 0x80) else b'\x00'
- return struct_unpack('<i', bs + last_byte)[0]
+ return compat_struct_unpack('<i', bs + last_byte)[0]
def _read_string(reader):
def _read_byte(reader):
resb = _read_bytes(1, reader=reader)
- res = struct_unpack('<B', resb)[0]
+ res = compat_struct_unpack('<B', resb)[0]
return res
print_notes(to_screen, versions_info['versions'])
- filename = sys.argv[0]
- # Py2EXE: Filename could be different
- if hasattr(sys, 'frozen') and not os.path.isfile(filename):
- if os.path.isfile(filename + '.exe'):
- filename += '.exe'
+ # sys.executable is set to the full pathname of the exe-file for py2exe
+ filename = sys.executable if hasattr(sys, 'frozen') else sys.argv[0]
if not os.access(filename, os.W_OK):
to_screen('ERROR: no write permissions on %s' % filename)
# Py2EXE
if hasattr(sys, 'frozen'):
- exe = os.path.abspath(filename)
+ exe = filename
directory = os.path.dirname(exe)
if not os.access(directory, os.W_OK):
to_screen('ERROR: no write permissions on %s' % directory)
import errno
import functools
import gzip
-import itertools
import io
+import itertools
import json
import locale
import math
import pipes
import platform
import re
-import ssl
import socket
-import struct
+import ssl
import subprocess
import sys
import tempfile
compat_http_client,
compat_kwargs,
compat_parse_qs,
+ compat_shlex_quote,
compat_socket_create_connection,
compat_str,
+ compat_struct_pack,
compat_urllib_error,
compat_urllib_parse,
compat_urllib_parse_urlencode,
compat_urllib_parse_urlparse,
+ compat_urllib_parse_unquote_plus,
compat_urllib_request,
compat_urlparse,
compat_xpath,
- shlex_quote,
)
+from .socks import (
+ ProxyType,
+ sockssocket,
+)
+
+
+def register_socks_protocols():
+ # "Register" SOCKS protocols
+ # In Python < 2.6.5, urlsplit() suffers from bug https://bugs.python.org/issue7904
+ # URLs with protocols not in urlparse.uses_netloc are not handled correctly
+ for scheme in ('socks', 'socks4', 'socks4a', 'socks5'):
+ if scheme not in compat_urlparse.uses_netloc:
+ compat_urlparse.uses_netloc.append(scheme)
+
# This is not clearly defined otherwise
compiled_regex_type = type(re.compile(''))
'wav',
'f4f', 'f4m', 'm3u8', 'smil')
+# needed for sanitizing filenames in restricted mode
+ACCENT_CHARS = dict(zip('ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ',
+ itertools.chain('AAAAAA', ['AE'], 'CEEEEIIIIDNOOOOOOO', ['OE'], 'UUUUUYP', ['ss'],
+ 'aaaaaa', ['ae'], 'ceeeeiiiionooooooo', ['oe'], 'uuuuuypy')))
+
def preferredencoding():
"""Get preferred encoding.
m = re.search(r'''(?xs)
<([a-zA-Z0-9:._-]+)
- (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]+|="[^"]+"|='[^']+'))*?
+ (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]*|="[^"]*"|='[^']*'))*?
\s+%s=['"]?%s['"]?
- (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]+|="[^"]+"|='[^']+'))*?
+ (?:\s+[a-zA-Z0-9:._-]+(?:=[a-zA-Z0-9:._-]*|="[^"]*"|='[^']*'))*?
\s*>
(?P<content>.*?)
</\1>
Set is_id if this is not an arbitrary string, but an ID that should be kept if possible
"""
def replace_insane(char):
+ if restricted and char in ACCENT_CHARS:
+ return ACCENT_CHARS[char]
if char == '?' or ord(char) < 32 or ord(char) == 127:
return ''
elif char == '"':
self._params = params
def http_open(self, req):
+ conn_class = compat_http_client.HTTPConnection
+
+ socks_proxy = req.headers.get('Ytdl-socks-proxy')
+ if socks_proxy:
+ conn_class = make_socks_conn_class(conn_class, socks_proxy)
+ del req.headers['Ytdl-socks-proxy']
+
return self.do_open(functools.partial(
- _create_http_connection, self, compat_http_client.HTTPConnection, False),
+ _create_http_connection, self, conn_class, False),
req)
@staticmethod
# As of RFC 2616 default charset is iso-8859-1 that is respected by python 3
if sys.version_info >= (3, 0):
location = location.encode('iso-8859-1').decode('utf-8')
+ else:
+ location = location.decode('utf-8')
location_escaped = escape_url(location)
if location != location_escaped:
del resp.headers['Location']
+ if sys.version_info < (3, 0):
+ location_escaped = location_escaped.encode('utf-8')
resp.headers['Location'] = location_escaped
return resp
https_response = http_response
+def make_socks_conn_class(base_class, socks_proxy):
+ assert issubclass(base_class, (
+ compat_http_client.HTTPConnection, compat_http_client.HTTPSConnection))
+
+ url_components = compat_urlparse.urlparse(socks_proxy)
+ if url_components.scheme.lower() == 'socks5':
+ socks_type = ProxyType.SOCKS5
+ elif url_components.scheme.lower() in ('socks', 'socks4'):
+ socks_type = ProxyType.SOCKS4
+ elif url_components.scheme.lower() == 'socks4a':
+ socks_type = ProxyType.SOCKS4A
+
+ def unquote_if_non_empty(s):
+ if not s:
+ return s
+ return compat_urllib_parse_unquote_plus(s)
+
+ proxy_args = (
+ socks_type,
+ url_components.hostname, url_components.port or 1080,
+ True, # Remote DNS
+ unquote_if_non_empty(url_components.username),
+ unquote_if_non_empty(url_components.password),
+ )
+
+ class SocksConnection(base_class):
+ def connect(self):
+ self.sock = sockssocket()
+ self.sock.setproxy(*proxy_args)
+ if type(self.timeout) in (int, float):
+ self.sock.settimeout(self.timeout)
+ self.sock.connect((self.host, self.port))
+
+ if isinstance(self, compat_http_client.HTTPSConnection):
+ if hasattr(self, '_context'): # Python > 2.6
+ self.sock = self._context.wrap_socket(
+ self.sock, server_hostname=self.host)
+ else:
+ self.sock = ssl.wrap_socket(self.sock)
+
+ return SocksConnection
+
+
class YoutubeDLHTTPSHandler(compat_urllib_request.HTTPSHandler):
def __init__(self, params, https_conn_class=None, *args, **kwargs):
compat_urllib_request.HTTPSHandler.__init__(self, *args, **kwargs)
def https_open(self, req):
kwargs = {}
+ conn_class = self._https_conn_class
+
if hasattr(self, '_context'): # python > 2.6
kwargs['context'] = self._context
if hasattr(self, '_check_hostname'): # python 3.x
kwargs['check_hostname'] = self._check_hostname
+
+ socks_proxy = req.headers.get('Ytdl-socks-proxy')
+ if socks_proxy:
+ conn_class = make_socks_conn_class(conn_class, socks_proxy)
+ del req.headers['Ytdl-socks-proxy']
+
return self.do_open(functools.partial(
- _create_http_connection, self, self._https_conn_class, True),
+ _create_http_connection, self, conn_class, True),
req, **kwargs)
format_expressions.extend([
'%d-%m-%Y',
'%d.%m.%Y',
+ '%d.%m.%y',
'%d/%m/%Y',
'%d/%m/%y',
'%d/%m/%Y %H:%M:%S',
if upload_date is None:
timetuple = email.utils.parsedate_tz(date_str)
if timetuple:
- upload_date = datetime.datetime(*timetuple[:6]).strftime('%Y%m%d')
+ try:
+ upload_date = datetime.datetime(*timetuple[:6]).strftime('%Y%m%d')
+ except ValueError:
+ pass
if upload_date is not None:
return compat_str(upload_date)
def intlist_to_bytes(xs):
if not xs:
return b''
- return struct_pack('%dB' % len(xs), *xs)
+ return compat_struct_pack('%dB' % len(xs), *xs)
# Cross-platform file locking
def remove_start(s, start):
- if s.startswith(start):
- return s[len(start):]
- return s
+ return s[len(start):] if s is not None and s.startswith(start) else s
def remove_end(s, end):
- if s.endswith(end):
- return s[:-len(end)]
- return s
+ return s[:-len(end)] if s is not None and s.endswith(end) else s
def remove_quotes(s):
fragment=escape_rfc3986(url_parsed.fragment)
).geturl()
-try:
- struct.pack('!I', 0)
-except TypeError:
- # In Python 2.6 and 2.7.x < 2.7.7, struct requires a bytes argument
- # See https://bugs.python.org/issue19099
- def struct_pack(spec, *args):
- if isinstance(spec, compat_str):
- spec = spec.encode('ascii')
- return struct.pack(spec, *args)
-
- def struct_unpack(spec, *args):
- if isinstance(spec, compat_str):
- spec = spec.encode('ascii')
- return struct.unpack(spec, *args)
-else:
- struct_pack = struct.pack
- struct_unpack = struct.unpack
-
def read_batch_urls(batch_fd):
def fixup(url):
def strip_jsonp(code):
return re.sub(
- r'(?s)^[a-zA-Z0-9_.]+\s*\(\s*(.*)\);?\s*?(?://[^\n]*)*$', r'\1', code)
+ r'(?s)^[a-zA-Z0-9_.$]+\s*\(\s*(.*)\);?\s*?(?://[^\n]*)*$', r'\1', code)
def js_to_json(code):
v = m.group(0)
if v in ('true', 'false', 'null'):
return v
- if v.startswith('"'):
- v = re.sub(r"\\'", "'", v[1:-1])
- elif v.startswith("'"):
- v = v[1:-1]
- v = re.sub(r"\\\\|\\'|\"", lambda m: {
- '\\\\': '\\\\',
- "\\'": "'",
+ elif v.startswith('/*') or v == ',':
+ return ""
+
+ if v[0] in ("'", '"'):
+ v = re.sub(r'(?s)\\.|"', lambda m: {
'"': '\\"',
- }[m.group(0)], v)
+ "\\'": "'",
+ '\\\n': '',
+ '\\x': '\\u00',
+ }.get(m.group(0), m.group(0)), v[1:-1])
+
+ INTEGER_TABLE = (
+ (r'^0[xX][0-9a-fA-F]+', 16),
+ (r'^0+[0-7]+', 8),
+ )
+
+ for regex, base in INTEGER_TABLE:
+ im = re.match(regex, v)
+ if im:
+ i = int(im.group(0), base)
+ return '"%d":' % i if v.endswith(':') else '%d' % i
+
return '"%s"' % v
- res = re.sub(r'''(?x)
- "(?:[^"\\]*(?:\\\\|\\['"nu]))*[^"\\]*"|
- '(?:[^'\\]*(?:\\\\|\\['"nu]))*[^'\\]*'|
- [a-zA-Z_][.a-zA-Z_0-9]*
+ return re.sub(r'''(?sx)
+ "(?:[^"\\]*(?:\\\\|\\['"nurtbfx/\n]))*[^"\\]*"|
+ '(?:[^'\\]*(?:\\\\|\\['"nurtbfx/\n]))*[^'\\]*'|
+ /\*.*?\*/|,(?=\s*[\]}])|
+ [a-zA-Z_][.a-zA-Z_0-9]*|
+ (?:0[xX][0-9a-fA-F]+|0+[0-7]+)(?:\s*:)?|
+ [0-9]+(?=\s*:)
''', fix_kv, code)
- res = re.sub(r',(\s*[\]}])', lambda m: m.group(1), res)
- return res
def qualities(quality_ids):
def args_to_str(args):
# Get a short string representation for a subprocess command
- return ' '.join(shlex_quote(a) for a in args)
+ return ' '.join(compat_shlex_quote(a) for a in args)
def error_to_compat_str(err):
def mimetype2ext(mt):
+ if mt is None:
+ return None
+
ext = {
'audio/mp4': 'm4a',
}.get(mt)
def urlhandle_detect_ext(url_handle):
- try:
- url_handle.headers
- getheader = lambda h: url_handle.headers[h]
- except AttributeError: # Python < 3
- getheader = url_handle.info().getheader
+ getheader = url_handle.headers.get
cd = getheader('Content-Disposition')
if cd:
if proxy == '__noproxy__':
return None # No Proxy
+ if compat_urlparse.urlparse(proxy).scheme.lower() in ('socks', 'socks4', 'socks4a', 'socks5'):
+ req.add_header('Ytdl-socks-proxy', proxy)
+ # youtube-dl's http/https handlers do wrapping the socket with socks
+ return None
return compat_urllib_request.ProxyHandler.proxy_open(
self, req, proxy, type)
from __future__ import unicode_literals
-__version__ = '2016.04.19'
+__version__ = '2016.06.03'
projects / youtube-dl / commitdiff