medup v0.1.8

Download all content from Medium to local folder
  • v0.1.8 - May 22, 2020
  • v0.1.7 - March 15, 2020
  • v0.1.6 - March 12, 2020
  • v0.1.5 - January 21, 2020
  • v0.1.4 - December 27, 2019

Logo

Medup

Sync all content from Medium with local folder via API

Table of Contents

Features

  • Discover all articles from user account available in public
  • Allow to download all recommended articles by user
  • Download images used inside article
  • Save posts in markdown format
  • Convert a single article in markdown format

Getting Started

Homebrew

Install medup tool to the MacOS or Linux via Hombrew:

$ brew tap miry/medup
$ brew install medup

Run export command for Medium author miry and save articles to local folder:

$ medup -u miry -d .

Docker

Docker way to make same job:

$ docker run -v <path to local articles folder>:/posts -it miry/medup -u <user>

Download all articles that user has recommended:

$ docker run -v <path to local articles folder>:/posts -it miry/medup -u <user> --recommended -d posts/recommended

Crystal

Run dumping with source code

$ shards install
$ crystal run src/cli.cr -- -u <medium user> -d <destination folder>

Example download all posts from author https://medium.com/@miry to local folder posts/miry

$ crystal run src/cli.cr -- -u miry -d posts/miry

Build

Build a application binary and execute:

$ shards install
$ rake build
$ _output/medup -u <medium user> -d <destination folder>

Result

By default all articles destination folder is posts.

In the directory, you can find 2 format of files: .json and .md.

  • JSON format is the raw, what Medium returns.
  • Markdown format is simple implementation of block formated text.

Images and IFRAME content are located in posts/assets.

Markdown

Example of exported document (Original Medium post vs Exported Markdown):

Contributing

  1. Fork it ( https://github.com/miry/medup/fork )
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a new Pull Request

Contributors

License

This project is under the LGPL-3.0 license.

Play ownself

Medium API

  1. Medium API documentation

  2. Generate token on the page https://medium.com/me/settings

  3. Create environment variable MEDIUM_TOKEN=<token>

  4. Verify token with sample query:

$ curl -H "Authorization: Bearer <token>" https://api.medium.com/v1/me
{"data":{"id":"number","username":"miry","name":"Michael Nikitochkin","url":"https://medium.com/@miry","imageUrl":"https://cdn-images-1.medium.com/fit/c/400/400/0*KgbjgGnH-csHuB8j."}}

Crawler

  1. Check public information not covered by API
$ curl "https://medium.com/@miry?format=json" | cut -c17- && : Remove in the front from response some strange JS code.
  1. Pagination
$ curl "https://medium.com/@miry/latest?format=json&limit=100" | cut -c17- && : Remove in the front from response some strange JS code.
  1. Post info
curl -s -H "Content-Type: application/json" https://medium.com/@miry/c35b40c499e\?format\=json\&limit\=100
  1. Stream
$ curl -s -H "Content-Type: application/json" "https://medium.com/_/api/users/fdf238948af6/profile/stream?source=overview" | cut -c17-
$ curl -s -H "Content-Type: application/json" "https://medium.com/_/api/users/fdf238948af6/profile/stream?limit=100&page=3" | cut -c17- > stream.json
$ cat stream.json| jq ".payload.references.Post[].title"
$ cat stream.json| jq ".payload.paging.next"
  1. Recommendations
$ curl -s -H "Content-Type: application/json" https://medium.com/@miry/has-recommended | cut -c17-
$ curl -s -H "Content-Type: application/json" "https://medium.com/_/api/users/fdf238948af6/profile/stream?source=has-recommended" | cut -c17-
Owner
github statistic
  • 9
  • 0
  • 5
  • 13
  • 10 days ago
  • December 1, 2019
License

GNU Lesser General Public License v3.0

Links
Synced at

Mon, 01 Jun 2020 13:55:59 GMT