mymodern
mymodern
mymodern is a command line program written in Crystal that scrapes web pages from public library websites. It uses CSS selector rules to parse the pages and lists checked-out books and books on hold ready for pickup.
The program uses Crystal's concurrency support to handle multiple HTTP requests simultaneously. It is modular, allowing you to define modules for different library websites and customize the scraping logic for each.
Installation
To set up mymodern on your machine:
- Clone the repository.
- Navigate to the project directory.
- Run the following commands:
shards install
shards build
The library kostya/lexbor parses HTML using CSS selectors. It wraps native C code and the compilation step for this dependency happens automatically when you run shards install
Once installed, the library will be available locally within your project directory.
For more information about the project and how to customize it for your needs, please refer to our:
-
Detailed README for detailed installation instructions, and subsequent setup and usage notes.
-
Project Structure Documentation which outlines the specific parts you'll need to adapt or modify to work with your library's website.
Usage
~/at_the_prompt/crystal/mymodern$ bin/mymodern --help
Usage: mymodern [OPTIONS]
Scrape pages at public libraries' web sites.
-m, --mock this option mocks everything
-t, --trace trace where it's all happening
-h, --help show this message
Example program output:
Hennepin Books Out
The Astronomer
Tuesday December 04, 2018
Subterranean Twin Cities
Wednesday December 05, 2018
Renewed: 1 time
2 people waiting
Hennepin Books on Hold
A History of America in Ten Strikes
Wednesday November 14, 2018
St. Paul Books Out
St. Paul Books on Hold
The mysterious flame of Queen Loana
Saturday November 17, 2018
Development
The program calls upon identically named methods provided by one or another library's module.
module Hennepin
def self.lib_data
# returns data in the form of a NamedTuple
def self.parse_checkedout_page(page)
# return a sorted array of CheckedOutRecord
# uses already existing record and comparison definitions
def self.parse_on_hold_page(page)
# return a sorted array of OnHoldRecord
Code for another library where you're a regular patron would do the same, and parts of the program that invoke these methods iterate over an array of the module names, and don't hard code the number or names of the modules.
You can adapt the modules that define these few methods, name them Springfield and Shelbyville, and require them from a single file which also lists them in an array of module names.
require "./hennepin"
require "./stpaul"
MODULE_NAMES = [Hennepin, StPaul]
So this can be used to scrape web pages at 1, 2, or 3 libraries, and all things being equal, runs in constant time no matter how many you're doing. *
* Because concurrency.
1 web site, 1.5 times faster. All things being equal, 2 websites, could be 3 times faster, 3 web sites, 4.5 times faster. Conceivably.
Contributing
- Fork the repository (https://github.com/Traven-B/mymodern/fork)
- Create a new feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add some feature'
) - Push to the branch (
git push origin my-new-feature
) - Create a new Pull Request
Contributors
- Traven-B Michael Kamb - creator, maintainer
mymodern
- 0
- 0
- 0
- 0
- 1
- 4 months ago
- June 28, 2024
MIT License
Mon, 25 Nov 2024 06:51:58 GMT