Python and Jupyter Notebooks

Python is a highly versatile and widely-used programming language, renowned for its readability and broad library support. Jupyter Notebooks, on the other hand, is an interactive computing environment that enables users to create and share documents containing live code, equations, visualizations, and narrative text. Together, they form a powerful toolkit for data analysis, scientific research, and educational purposes.

We will play with Python and Jupyter Notebooks to get a feel for both. This is a great interactive way to start development.

Emoji Print

It is easy to add an emoji to a message in code. However, using the emoji library or other libraries often requires you to install code on your machine. Before using a library, that is not part of Python distribution, you must install with pip

# terminal command to install library
$ pip install emoji
Collecting emoji
  Downloading emoji-2.5.1.tar.gz (356 kB)
...
Successfully installed emoji-2.5.1
!pip install emoji
from emoji import emojize 
print(emojize(":thumbs_up: Python is awesome! :grinning_face:"))
Defaulting to user installation because normal site-packages is not writeable
Collecting emoji
  Downloading emoji-2.12.1-py3-none-any.whl (431 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 431.4/431.4 KB 4.2 MB/s eta 0:00:00a 0:00:01
[?25hRequirement already satisfied: typing-extensions>=4.7.0 in /home/spooketti/.local/lib/python3.10/site-packages (from emoji) (4.9.0)
Installing collected packages: emoji
Successfully installed emoji-2.12.1
👍 Python is awesome! 😀

Extracting Data

Web sites become a lot more interesting when you are working with data, not trying to create it. Here is some code using a library called newspaper, this extracts a couple of writeups from the CNN Entertainment site.

!pip install newspaper3k
from newspaper import Article
from IPython.display import display, Markdown


urls = ["http://cnn.com/2023/03/29/entertainment/the-mandalorian-episode-5-recap/index.html", 
        "https://www.cnn.com/2023/06/09/entertainment/jurassic-park-anniversary/index.html"]

for url in urls:
    article = Article(url)
    article.download()
    article.parse()
    # Jupyter Notebook Display
    # print(article.title)
    display(Markdown(article.title)) # Jupyter display only
    display(Markdown(article.text)) # Jupyter display only
    print("\n")

Defaulting to user installation because normal site-packages is not writeable
Collecting newspaper3k
  Downloading newspaper3k-0.2.8-py3-none-any.whl (211 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 211.1/211.1 KB 3.5 MB/s eta 0:00:00a 0:00:01
[?25hCollecting feedparser>=5.2.1
  Downloading feedparser-6.0.11-py3-none-any.whl (81 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.3/81.3 KB 6.2 MB/s eta 0:00:00
[?25hRequirement already satisfied: beautifulsoup4>=4.4.1 in /usr/lib/python3/dist-packages (from newspaper3k) (4.10.0)
Collecting tldextract>=2.0.1
  Downloading tldextract-5.1.2-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.6/97.6 KB 4.6 MB/s eta 0:00:00
[?25hCollecting feedfinder2>=0.0.4
  Downloading feedfinder2-0.0.4.tar.gz (3.3 kB)
  Preparing metadata (setup.py) ... [?25ldone
[?25hCollecting jieba3k>=0.35.1
  Downloading jieba3k-0.35.1.zip (7.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.4/7.4 MB 13.7 MB/s eta 0:00:0000:0100:01
[?25h  Preparing metadata (setup.py) ... [?25ldone
[?25hRequirement already satisfied: Pillow>=3.3.0 in /usr/lib/python3/dist-packages (from newspaper3k) (9.0.1)
Collecting nltk>=3.2.1
  Downloading nltk-3.9.1-py3-none-any.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 24.8 MB/s eta 0:00:00a 0:00:01
[?25hRequirement already satisfied: cssselect>=0.9.2 in /home/spooketti/.local/lib/python3.10/site-packages (from newspaper3k) (1.2.0)
Requirement already satisfied: requests>=2.10.0 in /home/spooketti/.local/lib/python3.10/site-packages (from newspaper3k) (2.31.0)
Requirement already satisfied: PyYAML>=3.11 in /usr/lib/python3/dist-packages (from newspaper3k) (5.4.1)
Requirement already satisfied: lxml>=3.6.0 in /usr/lib/python3/dist-packages (from newspaper3k) (4.8.0)
Requirement already satisfied: python-dateutil>=2.5.3 in /home/spooketti/.local/lib/python3.10/site-packages (from newspaper3k) (2.8.2)
Collecting tinysegmenter==0.3
  Downloading tinysegmenter-0.3.tar.gz (16 kB)
  Preparing metadata (setup.py) ... [?25ldone
[?25hRequirement already satisfied: six in /usr/lib/python3/dist-packages (from feedfinder2>=0.0.4->newspaper3k) (1.16.0)
Collecting sgmllib3k
  Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB)
  Preparing metadata (setup.py) ... [?25ldone
[?25hRequirement already satisfied: tqdm in /home/spooketti/.local/lib/python3.10/site-packages (from nltk>=3.2.1->newspaper3k) (4.66.2)
Requirement already satisfied: joblib in /home/spooketti/.local/lib/python3.10/site-packages (from nltk>=3.2.1->newspaper3k) (1.3.2)
Collecting regex>=2021.8.3
  Downloading regex-2024.7.24-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (776 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 776.5/776.5 KB 22.4 MB/s eta 0:00:0000:01
[?25hRequirement already satisfied: click in /home/spooketti/.local/lib/python3.10/site-packages (from nltk>=3.2.1->newspaper3k) (8.1.7)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/spooketti/.local/lib/python3.10/site-packages (from requests>=2.10.0->newspaper3k) (3.3.0)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/spooketti/.local/lib/python3.10/site-packages (from requests>=2.10.0->newspaper3k) (1.26.18)
Requirement already satisfied: idna<4,>=2.5 in /home/spooketti/.local/lib/python3.10/site-packages (from requests>=2.10.0->newspaper3k) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/spooketti/.local/lib/python3.10/site-packages (from requests>=2.10.0->newspaper3k) (2023.7.22)
Collecting requests-file>=1.4
  Downloading requests_file-2.1.0-py2.py3-none-any.whl (4.2 kB)
Collecting filelock>=3.0.8
  Downloading filelock-3.16.0-py3-none-any.whl (16 kB)
Building wheels for collected packages: tinysegmenter, feedfinder2, jieba3k, sgmllib3k
  Building wheel for tinysegmenter (setup.py) ... [?25ldone
[?25h  Created wheel for tinysegmenter: filename=tinysegmenter-0.3-py3-none-any.whl size=13553 sha256=72645567617d886b8e3ecc094ebfd7f7477d8440bd6c8c4a2f5fbd3336c22f72
  Stored in directory: /home/spooketti/.cache/pip/wheels/c8/d6/6c/384f58df48c00b9a31d638005143b5b3ac62c3d25fb1447f23
  Building wheel for feedfinder2 (setup.py) ... [?25ldone
[?25h  Created wheel for feedfinder2: filename=feedfinder2-0.0.4-py3-none-any.whl size=3357 sha256=1e3fa1a2da46e14c2f4428af5b32a68206062c69552fd41f01ac7707422da632
  Stored in directory: /home/spooketti/.cache/pip/wheels/97/02/e7/a1ff1760e12bdbaab0ac824fae5c1bc933e41c4ccd6a8f8edb
  Building wheel for jieba3k (setup.py) ... [?25ldone
[?25h  Created wheel for jieba3k: filename=jieba3k-0.35.1-py3-none-any.whl size=7398404 sha256=c7a5927759bb0f655dd76487dcd9940a948b80bdf28dabac4042a98986247891
  Stored in directory: /home/spooketti/.cache/pip/wheels/7a/c4/0c/12a9a314ecac499456c4c3b2fcc2f635a3b45a39dfbd240299
  Building wheel for sgmllib3k (setup.py) ... [?25ldone
[?25h  Created wheel for sgmllib3k: filename=sgmllib3k-1.0.0-py3-none-any.whl size=6066 sha256=361e47ae11f4cc0a200df078d6910e6d6cacf6637b923b1c861cb5c00ce4c573
  Stored in directory: /home/spooketti/.cache/pip/wheels/f0/69/93/a47e9d621be168e9e33c7ce60524393c0b92ae83cf6c6e89c5
Successfully built tinysegmenter feedfinder2 jieba3k sgmllib3k
Installing collected packages: tinysegmenter, sgmllib3k, jieba3k, regex, filelock, feedparser, requests-file, nltk, feedfinder2, tldextract, newspaper3k
Successfully installed feedfinder2-0.0.4 feedparser-6.0.11 filelock-3.16.0 jieba3k-0.35.1 newspaper3k-0.2.8 nltk-3.9.1 regex-2024.7.24 requests-file-2.1.0 sgmllib3k-1.0.0 tinysegmenter-0.3 tldextract-5.1.2

‘The Mandalorian’ finally comes into focus, while giving out a ‘Rebels’ yell

Editor’s Note: The following contains spoilers about the fifth episode of “The Mandalorian,” Season 3, “The Pirate.”

CNN —

After what can at best be described as a somewhat disjointed third season thus far, the fifth episode of “The Mandalorian” began to bring those pieces together and into focus, while continuing to draw upon the “Star Wars” animated series that preceded it, including another cameo by a character from the rightfully lauded “Rebels.”

Subtitled “The Pirate,” the episode presented further evidence of the dysfunctional nature of the New Republic, unable or unwilling to defend a faraway planet from an invading band of pirates. (Lucasfilm being a unit of Disney, the marauders had a certain “Yo ho, yo ho” vibe to them.)

The siege also played into Mandalorian politics, and the efforts of Bo-Katan (Katee Sackhoff) to reclaim her heritage and potentially reunite her people’s various tribes, after leading them, along with Din Djarin (voiced by Pedro Pascal), to the rescue of his pal Greef Karga (Carl Weathers) and the planet’s residents.

Still, the most pleasing moment for longtime “Star Wars” fans was likely what amounted to a throwaway scene, introducing a live-action version of the hulking alien Zeb, a character from the animated “Star Wars Rebels,” which concluded in 2018. “The Mandalorian” has drawn heavily from those properties, which were overseen by one of its executive producers, Dave Filoni. (In another nice touch, Steve Blum again provided the voice of the character, and Zeb looked a whole lot better than the pirate leader.)

Finally, the episode closed with evidence that the evil Moff Gideon (played by Giancarlo Esposito previously) had seemingly been freed from the prison ship that was transporting him to stand trial, reviving that potential threat.

Having resolved the fate of Grogu, a.k.a. Baby Yoda, during the first two seasons, “The Mandalorian” has thus moved on to fill in narrative gaps about an under-explored chapter in “Star Wars” history – namely, the factors that resulted in the fall of the New Republic and the rise of the First Order, the plot line featured in the most recent trilogy that began with “The Force Awakens.”

“This isn’t a rebellion anymore,” a bureaucrat (played by Tim Meadows) says about what happens outside of the New Republic’s jurisdiction, conveying an indifference to the fate of the planet Nevarro overtly articulated later when it was observed that the governing body in Coruscant “doesn’t care.”

Executive producers Jon Favreau and Filoni have taken their time in reaching this point, juggling these various issues in somewhat ungainly fashion through the first half of the season. That perhaps reflects the transition of the show to an emphasis on the macro instead of the micro, while still finding time to detour for the occasional “Rebels” yell.

‘Jurassic Park’ still has bite at 30 years old, and here’s why

CNN —

It’s been 30 years since Steven Spielberg’s dinosaurs stampeded across the screen in the first “Jurassic Park,” but it feels more recent.

I was 12 in June of 1993 and vividly remember watching with glee when the Tyrannosaurus Rex, with its teeny arms and perpetual scowl, blew the walls of the bathroom down like a big bad wolf and promptly ate the lawyer character (played to hilarious effect by Martin Ferrero). Part of this, surely, had to do with the fact that I was a mouthy pre-teen, and many adults in my sphere at the time opined that I “would make a great lawyer” just like my father – a fate I abhorred.

Admittedly, I was the exact target audience for this creature feature, and even though I was already somewhat of a self-taught critic (note the aforementioned mouthiness), I was awed by what I saw that summer three decades ago, and my impressions of “Jurassic Park” remain to this day.

Joseph Mazzello in “Jurassic Park.” Amblin/Universal/Kobal/Shutterstock

Part of that lasting impact, of course, has to do with the still-groundbreaking effects in the movie, which surprisingly hold up, and on a fairly hi-tech 72-inch TV screen to boot. While the first dino money shot – of the plant-eating brachiosaurus – might look just a tad soupy in 2023, it still looks considerably better than more contemporary fare, and the ensuing imagery of the more predatory beasts (like T-rex and especially those raptors) remains top-notch. The computer-generated imagery in the movie is essentially credited with marking the end of stop-motion animation as the go-to effects option for films such as these, notably used in everything that came before, from 1933’s “King Kong” to 1981’s “Clash of the Titans.” The animatronics are something to behold as well, particularly the ailing triceratops responsible for that “one big pile of s—,” one of many priceless quips uttered by Ian Malcolm (Jeff Goldblum).

The appeal of “Jurassic,” based on Michael Crichton’s acclaimed novel, is also largely due to the film’s suspenseful and pared-down pacing, which of course can be linked to Spielberg, who learned a thing or two about keeping his cards close to his chest with “Jaws” – the great white mother of all creature features that famously showed startlingly little of the big fish before the climax.

Another “Jaws” connection is prolific film composer John Williams, the Spielberg collaborator who created a majestic score for “Jurassic Park” that is still synonymous with an air of discovery, one that can easily be hummed when looking upon any great view or upon entering a new and uncharted space.

Laura Dern, Sam Neill and Joseph Mazzello in “Jurassic Park.” Amblin/Universal/Kobal/Shutterstock

And then there’s the casting, an element that sometimes takes a number of years to truly appreciate. Aside from the always-dependable Goldblum, there’s Laura Dern, who carved out her own Sigourney Weaver-shaped notch in the movie thanks to that one terrifying sequence in the control shed. Plus, her reaction shot to that first dinosaur reveal – along with that of Sam Neill – could be viewed as a textbook for green-screen acting, which has become the standard ever since, in Marvel movies and beyond. Add to that the amazing and meme-worthy smaller performances from Samuel L. Jackson (“Hold onto your butts!”), Wayne Knight (“Ah ah ah! You didn’t say the magic word!”) and Bob Peck (“Clever girl”), and you’ve got a crowd-pleaser that is equal parts adventure, comedy and chomp-chomp thriller.

While the rest of the entries in the “Jurassic” franchise have not exactly been up to par (aside from 2015’s not-terrible first reboot “Jurassic World”), the original flick still “rules” – and is definitely worth a rewatch on the occasion of its 30th birthday.

#!pip install wikipedia
import wikipedia 
from IPython.display import display, Markdown # add for Jupyter

terms = ["Python (programming language)", "JavaScript"]
for term in terms:
    # Search for a page 
    result = wikipedia.search(term)
    # Get the summary of the first result
    summary = wikipedia.summary(result[0])
    print(term) 
    # print(summary) # console display
    display(Markdown(summary)) # Jupyter display
Python (programming language)

Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation. Python is dynamically typed and garbage-collected. It supports multiple programming paradigms, including structured (particularly procedural), object-oriented and functional programming. It is often described as a “batteries included” language due to its comprehensive standard library. Guido van Rossum began working on Python in the late 1980s as a successor to the ABC programming language and first released it in 1991 as Python 0.9.0. Python 2.0 was released in 2000. Python 3.0, released in 2008, was a major revision not completely backward-compatible with earlier versions. Python 2.7.18, released in 2020, was the last release of Python 2. Python consistently ranks as one of the most popular programming languages, and has gained widespread use in the machine learning community.

JavaScript

JavaScript (), often abbreviated as JS, is a programming language and core technology of the Web, alongside HTML and CSS. 99% of websites use JavaScript on the client side for webpage behavior. Web browsers have a dedicated JavaScript engine that executes the client code. These engines are also utilized in some servers and a variety of apps. The most popular runtime system for non-browser usage is Node.js. JavaScript is a high-level, often just-in-time compiled language that conforms to the ECMAScript standard. It has dynamic typing, prototype-based object-orientation, and first-class functions. It is multi-paradigm, supporting event-driven, functional, and imperative programming styles. It has application programming interfaces (APIs) for working with text, dates, regular expressions, standard data structures, and the Document Object Model (DOM). The ECMAScript standard does not include any input/output (I/O), such as networking, storage, or graphics facilities. In practice, the web browser or other runtime system provides JavaScript APIs for I/O. Although Java and JavaScript are similar in name, syntax, and respective standard libraries, the two languages are distinct and differ greatly in design.

Inspecting a Function

The inspect module can give you the output of what’s inside many Python functions/objects. This can help you explore code behind what you are using.

import inspect 
from newspaper import Article

# inspect newspaper Article function
print(inspect.getsource(Article))
class Article(object):
    """Article objects abstract an online news article page
    """
    def __init__(self, url, title='', source_url='', config=None, **kwargs):
        """The **kwargs argument may be filled with config values, which
        is added into the config object
        """
        if isinstance(title, Configuration) or \
                isinstance(source_url, Configuration):
            raise ArticleException(
                'Configuration object being passed incorrectly as title or '
                'source_url! Please verify `Article`s __init__() fn.')

        self.config = config or Configuration()
        self.config = extend_config(self.config, kwargs)

        self.extractor = ContentExtractor(self.config)

        if source_url == '':
            scheme = urls.get_scheme(url)
            if scheme is None:
                scheme = 'http'
            source_url = scheme + '://' + urls.get_domain(url)

        if source_url is None or source_url == '':
            raise ArticleException('input url bad format')

        # URL to the main page of the news source which owns this article
        self.source_url = source_url

        self.url = urls.prepare_url(url, self.source_url)

        self.title = title

        # URL of the "best image" to represent this article
        self.top_img = self.top_image = ''

        # stores image provided by metadata
        self.meta_img = ''

        # All image urls in this article
        self.imgs = self.images = []

        # All videos in this article: youtube, vimeo, etc
        self.movies = []

        # Body text from this article
        self.text = ''

        # `keywords` are extracted via nlp() from the body text
        self.keywords = []

        # `meta_keywords` are extracted via parse() from <meta> tags
        self.meta_keywords = []

        # `tags` are also extracted via parse() from <meta> tags
        self.tags = set()

        # List of authors who have published the article, via parse()
        self.authors = []

        self.publish_date = ''

        # Summary generated from the article's body txt
        self.summary = ''

        # This article's unchanged and raw HTML
        self.html = ''

        # The HTML of this article's main node (most important part)
        self.article_html = ''

        # Keep state for downloads and parsing
        self.is_parsed = False
        self.download_state = ArticleDownloadState.NOT_STARTED
        self.download_exception_msg = None

        # Meta description field in the HTML source
        self.meta_description = ""

        # Meta language field in HTML source
        self.meta_lang = ""

        # Meta favicon field in HTML source
        self.meta_favicon = ""

        # Meta tags contain a lot of structured data, e.g. OpenGraph
        self.meta_data = {}

        # The canonical link of this article if found in the meta data
        self.canonical_link = ""

        # Holds the top element of the DOM that we determine is a candidate
        # for the main body of the article
        self.top_node = None

        # A deepcopied clone of the above object before heavy parsing
        # operations, useful for users to query data in the
        # "most important part of the page"
        self.clean_top_node = None

        # lxml DOM object generated from HTML
        self.doc = None

        # A deepcopied clone of the above object before undergoing heavy
        # cleaning operations, serves as an API if users need to query the DOM
        self.clean_doc = None

        # A property dict for users to store custom data.
        self.additional_data = {}

    def build(self):
        """Build a lone article from a URL independent of the source (newspaper).
        Don't normally call this method b/c it's good to multithread articles
        on a source (newspaper) level.
        """
        self.download()
        self.parse()
        self.nlp()

    def download(self, input_html=None, title=None, recursion_counter=0):
        """Downloads the link's HTML content, don't use if you are batch async
        downloading articles

        recursion_counter (currently 1) stops refreshes that are potentially
        infinite
        """
        if input_html is None:
            try:
                html = network.get_html_2XX_only(self.url, self.config)
            except requests.exceptions.RequestException as e:
                self.download_state = ArticleDownloadState.FAILED_RESPONSE
                self.download_exception_msg = str(e)
                log.debug('Download failed on URL %s because of %s' %
                          (self.url, self.download_exception_msg))
                return
        else:
            html = input_html

        if self.config.follow_meta_refresh:
            meta_refresh_url = extract_meta_refresh(html)
            if meta_refresh_url and recursion_counter < 1:
                return self.download(
                    input_html=network.get_html(meta_refresh_url),
                    recursion_counter=recursion_counter + 1)

        self.set_html(html)
        self.set_title(title)

    def parse(self):
        self.throw_if_not_downloaded_verbose()

        self.doc = self.config.get_parser().fromstring(self.html)
        self.clean_doc = copy.deepcopy(self.doc)

        if self.doc is None:
            # `parse` call failed, return nothing
            return

        # TODO: Fix this, sync in our fix_url() method
        parse_candidate = self.get_parse_candidate()
        self.link_hash = parse_candidate.link_hash  # MD5

        document_cleaner = DocumentCleaner(self.config)
        output_formatter = OutputFormatter(self.config)

        title = self.extractor.get_title(self.clean_doc)
        self.set_title(title)

        authors = self.extractor.get_authors(self.clean_doc)
        self.set_authors(authors)

        meta_lang = self.extractor.get_meta_lang(self.clean_doc)
        self.set_meta_language(meta_lang)

        if self.config.use_meta_language:
            self.extractor.update_language(self.meta_lang)
            output_formatter.update_language(self.meta_lang)

        meta_favicon = self.extractor.get_favicon(self.clean_doc)
        self.set_meta_favicon(meta_favicon)

        meta_description = \
            self.extractor.get_meta_description(self.clean_doc)
        self.set_meta_description(meta_description)

        canonical_link = self.extractor.get_canonical_link(
            self.url, self.clean_doc)
        self.set_canonical_link(canonical_link)

        tags = self.extractor.extract_tags(self.clean_doc)
        self.set_tags(tags)

        meta_keywords = self.extractor.get_meta_keywords(
            self.clean_doc)
        self.set_meta_keywords(meta_keywords)

        meta_data = self.extractor.get_meta_data(self.clean_doc)
        self.set_meta_data(meta_data)

        self.publish_date = self.extractor.get_publishing_date(
            self.url,
            self.clean_doc)

        # Before any computations on the body, clean DOM object
        self.doc = document_cleaner.clean(self.doc)

        self.top_node = self.extractor.calculate_best_node(self.doc)
        if self.top_node is not None:
            video_extractor = VideoExtractor(self.config, self.top_node)
            self.set_movies(video_extractor.get_videos())

            self.top_node = self.extractor.post_cleanup(self.top_node)
            self.clean_top_node = copy.deepcopy(self.top_node)

            text, article_html = output_formatter.get_formatted(
                self.top_node)
            self.set_article_html(article_html)
            self.set_text(text)

        self.fetch_images()

        self.is_parsed = True
        self.release_resources()

    def fetch_images(self):
        if self.clean_doc is not None:
            meta_img_url = self.extractor.get_meta_img_url(
                self.url, self.clean_doc)
            self.set_meta_img(meta_img_url)

            imgs = self.extractor.get_img_urls(self.url, self.clean_doc)
            if self.meta_img:
                imgs.add(self.meta_img)
            self.set_imgs(imgs)

        if self.clean_top_node is not None and not self.has_top_image():
            first_img = self.extractor.get_first_img_url(
                self.url, self.clean_top_node)
            if self.config.fetch_images:
                self.set_top_img(first_img)
            else:
                self.set_top_img_no_check(first_img)

        if not self.has_top_image() and self.config.fetch_images:
            self.set_reddit_top_img()

    def has_top_image(self):
        return self.top_img is not None and self.top_img != ''

    def is_valid_url(self):
        """Performs a check on the url of this link to determine if article
        is a real news article or not
        """
        return urls.valid_url(self.url)

    def is_valid_body(self):
        """If the article's body text is long enough to meet
        standard article requirements, keep the article
        """
        if not self.is_parsed:
            raise ArticleException('must parse article before checking \
                                    if it\'s body is valid!')
        meta_type = self.extractor.get_meta_type(self.clean_doc)
        wordcount = self.text.split(' ')
        sentcount = self.text.split('.')

        if (meta_type == 'article' and len(wordcount) >
                (self.config.MIN_WORD_COUNT)):
            log.debug('%s verified for article and wc' % self.url)
            return True

        if not self.is_media_news() and not self.text:
            log.debug('%s caught for no media no text' % self.url)
            return False

        if self.title is None or len(self.title.split(' ')) < 2:
            log.debug('%s caught for bad title' % self.url)
            return False

        if len(wordcount) < self.config.MIN_WORD_COUNT:
            log.debug('%s caught for word cnt' % self.url)
            return False

        if len(sentcount) < self.config.MIN_SENT_COUNT:
            log.debug('%s caught for sent cnt' % self.url)
            return False

        if self.html is None or self.html == '':
            log.debug('%s caught for no html' % self.url)
            return False

        log.debug('%s verified for default true' % self.url)
        return True

    def is_media_news(self):
        """If the article is related heavily to media:
        gallery, video, big pictures, etc
        """
        safe_urls = ['/video', '/slide', '/gallery', '/powerpoint',
                     '/fashion', '/glamour', '/cloth']
        for s in safe_urls:
            if s in self.url:
                return True
        return False

    def nlp(self):
        """Keyword extraction wrapper
        """
        self.throw_if_not_downloaded_verbose()
        self.throw_if_not_parsed_verbose()

        nlp.load_stopwords(self.config.get_language())
        text_keyws = list(nlp.keywords(self.text).keys())
        title_keyws = list(nlp.keywords(self.title).keys())
        keyws = list(set(title_keyws + text_keyws))
        self.set_keywords(keyws)

        max_sents = self.config.MAX_SUMMARY_SENT

        summary_sents = nlp.summarize(title=self.title, text=self.text, max_sents=max_sents)
        summary = '\n'.join(summary_sents)
        self.set_summary(summary)

    def get_parse_candidate(self):
        """A parse candidate is a wrapper object holding a link hash of this
        article and a final_url of the article
        """
        if self.html:
            return RawHelper.get_parsing_candidate(self.url, self.html)
        return URLHelper.get_parsing_candidate(self.url)

    def build_resource_path(self):
        """Must be called after computing HTML/final URL
        """
        res_path = self.get_resource_path()
        if not os.path.exists(res_path):
            os.mkdir(res_path)

    def get_resource_path(self):
        """Every article object has a special directory to store data in from
        initialization to garbage collection
        """
        res_dir_fn = 'article_resources'
        resource_directory = os.path.join(settings.TOP_DIRECTORY, res_dir_fn)
        if not os.path.exists(resource_directory):
            os.mkdir(resource_directory)
        dir_path = os.path.join(resource_directory, '%s_' % self.link_hash)
        return dir_path

    def release_resources(self):
        # TODO: implement in entirety
        path = self.get_resource_path()
        for fname in glob.glob(path):
            try:
                os.remove(fname)
            except OSError:
                pass
        # os.remove(path)

    def set_reddit_top_img(self):
        """Wrapper for setting images. Queries known image attributes
        first, then uses Reddit's image algorithm as a fallback.
        """
        try:
            s = images.Scraper(self)
            self.set_top_img(s.largest_image_url())
        except TypeError as e:
            if "Can't convert 'NoneType' object to str implicitly" in e.args[0]:
                log.debug('No pictures found. Top image not set, %s' % e)
            elif 'timed out' in e.args[0]:
                log.debug('Download of picture timed out. Top image not set, %s' % e)
            else:
                log.critical('TypeError other than None type error. '
                             'Cannot set top image using the Reddit '
                             'algorithm. Possible error with PIL., %s' % e)
        except Exception as e:
            log.critical('Other error with setting top image using the '
                         'Reddit algorithm. Possible error with PIL, %s' % e)

    def set_title(self, input_title):
        if input_title:
            self.title = input_title[:self.config.MAX_TITLE]

    def set_text(self, text):
        text = text[:self.config.MAX_TEXT]
        if text:
            self.text = text

    def set_html(self, html):
        """Encode HTML before setting it
        """
        if html:
            if isinstance(html, bytes):
                html = self.config.get_parser().get_unicode_html(html)
            self.html = html
            self.download_state = ArticleDownloadState.SUCCESS

    def set_article_html(self, article_html):
        """Sets the HTML of just the article's `top_node`
        """
        if article_html:
            self.article_html = article_html

    def set_meta_img(self, src_url):
        self.meta_img = src_url
        self.set_top_img_no_check(src_url)

    def set_top_img(self, src_url):
        if src_url is not None:
            s = images.Scraper(self)
            if s.satisfies_requirements(src_url):
                self.set_top_img_no_check(src_url)

    def set_top_img_no_check(self, src_url):
        """Provide 2 APIs for images. One at "top_img", "imgs"
        and one at "top_image", "images"
        """
        self.top_img = src_url
        self.top_image = src_url

    def set_imgs(self, imgs):
        """The motive for this method is the same as above, provide APIs
        for both `article.imgs` and `article.images`
        """
        self.images = imgs
        self.imgs = imgs

    def set_keywords(self, keywords):
        """Keys are stored in list format
        """
        if not isinstance(keywords, list):
            raise Exception("Keyword input must be list!")
        if keywords:
            self.keywords = keywords[:self.config.MAX_KEYWORDS]

    def set_authors(self, authors):
        """Authors are in ["firstName lastName", "firstName lastName"] format
        """
        if not isinstance(authors, list):
            raise Exception("authors input must be list!")
        if authors:
            self.authors = authors[:self.config.MAX_AUTHORS]

    def set_summary(self, summary):
        """Summary here refers to a paragraph of text from the
        title text and body text
        """
        self.summary = summary[:self.config.MAX_SUMMARY]

    def set_meta_language(self, meta_lang):
        """Save langauges in their ISO 2-character form
        """
        if meta_lang and len(meta_lang) >= 2 and \
           meta_lang in get_available_languages():
            self.meta_lang = meta_lang[:2]

    def set_meta_keywords(self, meta_keywords):
        """Store the keys in list form
        """
        self.meta_keywords = [k.strip() for k in meta_keywords.split(',')]

    def set_meta_favicon(self, meta_favicon):
        self.meta_favicon = meta_favicon

    def set_meta_description(self, meta_description):
        self.meta_description = meta_description

    def set_meta_data(self, meta_data):
        self.meta_data = meta_data

    def set_canonical_link(self, canonical_link):
        self.canonical_link = canonical_link

    def set_tags(self, tags):
        self.tags = tags

    def set_movies(self, movie_objects):
        """Trim video objects into just urls
        """
        movie_urls = [o.src for o in movie_objects if o and o.src]
        self.movies = movie_urls

    def throw_if_not_downloaded_verbose(self):
        """Parse ArticleDownloadState -> log readable status
        -> maybe throw ArticleException
        """
        if self.download_state == ArticleDownloadState.NOT_STARTED:
            raise ArticleException('You must `download()` an article first!')
        elif self.download_state == ArticleDownloadState.FAILED_RESPONSE:
            raise ArticleException('Article `download()` failed with %s on URL %s' %
                  (self.download_exception_msg, self.url))

    def throw_if_not_parsed_verbose(self):
        """Parse `is_parsed` status -> log readable status
        -> maybe throw ArticleException
        """
        if not self.is_parsed:
            raise ArticleException('You must `parse()` an article first!')

Python Data Types

Dynamic typing means that the type of the variable is determined only during runtime. Strong typing means that variables do have a type and that the type matters when performing operations. In the illustration below there are two functions

  • mean… shows types required prior to calling average function
  • average, average2… calculates the average of a list of numbers

Python has types. In the language you can use type hints, but most coders do not use them. In other languages like Java and ‘C’ you must specify types.

import sys
from typing import Union

# Define types for mean function, trying to analyze input possibilities
Number = Union[int, float]  # Number can be either int or float type
Numbers = list[Number] # Numbers is a list of Number types
Scores = Union[Number, Numbers] # Scores can be single or multiple 

def mean(scores: Scores, method: int = 1) -> float:
    """
    Calculate the mean of a list of scores.
    
    Average and Average2 are hidden functions performing mean algorithm

    If a single score is provided in scores, it is returned as the mean.
    If a list of scores is provided, the average is calculated and returned.
    """
    
    def average(scores): 
        """Calculate the average of a list of scores using a Python for loop with rounding."""
        sum = 0
        len = 0
        for score in scores:
            if isinstance(score, Number):
                sum += score
                len += 1
            else:
                print("Bad data: " + str(score) + " in " + str(scores))
                sys.exit()
        return sum / len
    
    def average2(scores):
        """Calculate the average of a list of scores using the built-in sum() function with rounding."""
        return sum(scores) / len(scores)

    # test to see if scores is  a list of numbers
    if isinstance(scores, list):
        if method == 1:  
            # long method
            result = average(scores)
        else:
            # built in method
            result = average2(scores)
        return round(result + 0.005, 2)
    
    return scores # case where scores is a single valu

# try with one number
singleScore = 100
print("Print test data: " + str(singleScore))  # concat data for single line
print("Mean of single number: " + str(mean(singleScore)))

print()

# define a list of numbers
testScores = [90.5, 100, 85.4, 88]
print("Print test data: " + str(testScores))
print("Average score, loop method: " + str(mean(testScores)))
print("Average score, function method: " +  str(mean(testScores, 2)))

print()

badData = [100, "NaN", 90]
print("Print test data: " + str(badData))
print("Mean with bad data: " + str(mean(badData)))


Print test data: 100
Mean of single number: 100

Print test data: [90.5, 100, 85.4, 88]
Average score, loop method: 90.98
Average score, function method: 90.98

Print test data: [100, 'NaN', 90]
Bad data: NaN in [100, 'NaN', 90]



An exception has occurred, use %tb to see the full traceback.


SystemExit



/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py:3465: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

Hacks

Here is a summary of some of the things learned above.

  • Formatting messages with emoji
  • Exploring data with newspaper and wikipedia libraries
  • Finding code on how the library we used was made
  • Learning about data types while writing an algorithm for mean

Part of Project Based learning is the idea of combining concepts to form something more interesting. Make a plan, form some ideas, brainstorm ideas with pair. Produce something that is interesting and challenging. Samples…

Sentiment Analysis with Emojis

from textblob import TextBlob

sentence = "i really love dogs"
analysis = TextBlob(sentence).sentiment
print(analysis)
Sentiment(polarity=0.5, subjectivity=0.6)
import wikipedia 

term = str(input())
results = wikipedia.search(term)
if not results:
    raise Exception("No Articles Found")

def summarizeArticle(article):
    summary = wikipedia.summary(article)
    print(summary)

for result in results:
    try: 
        summarizeArticle(result)
        break
    except:
        pass
A steamboat is a boat that is propelled primarily by steam power, typically driving propellers or paddlewheels. Steamboats sometimes use the prefix designation SS, S.S. or S/S (for 'Screw Steamer') or PS (for 'Paddle Steamer'); however, these designations are most often used for steamships.
The term steamboat is used to refer to smaller, insular, steam-powered boats working on lakes and rivers, particularly riverboats. As using steam became more reliable, steam power became applied to larger, ocean-going vessels.
import wikipedia 
from textblob import TextBlob

term = str(input())


results = wikipedia.search(term)
if not results:
    raise Exception("No Articles Found")

def summarizeArticle(article):
    summary = wikipedia.summary(article)
    analysis = TextBlob(summary).sentiment
    print(summary)
    print(analysis)
    

for result in results:
    try: 
        summarizeArticle(result)
        break
    except:
        pass
/usr/lib/python3/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.4
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"


A steamboat is a boat that is propelled primarily by steam power, typically driving propellers or paddlewheels. Steamboats sometimes use the prefix designation SS, S.S. or S/S (for 'Screw Steamer') or PS (for 'Paddle Steamer'); however, these designations are most often used for steamships.
The term steamboat is used to refer to smaller, insular, steam-powered boats working on lakes and rivers, particularly riverboats. As using steam became more reliable, steam power became applied to larger, ocean-going vessels.


0.19999999999999998
Sentiment(polarity=0.19999999999999998, subjectivity=0.4761904761904762)