A powerful Streamlit web application that intelligently scrapes and summarizes articles from the Google AI Blog, featuring automated content processing, image management, and AI-powered insights.

Smart Collection Automatically discovers and extracts article links from Google AI Blog with intelligent parsing

AI-Powered Processing Generates comprehensive summaries using local Ollama models for enhanced content understanding

Interactive Dashboard Browse, search, and filter content through an intuitive Streamlit interface

• 📄 Content Processing → Downloads and structures article content with metadata
• 🖼️ Image Management → Automatically retrieves and organizes article images
• ⚙️ Flexible Configuration → Easy setup through YAML configuration files
• 🔍 Advanced Search → Powerful filtering and discovery tools
• 📊 Content Analytics → Insights into your scraped content library

content-inspiration/
│
├── 📋 config/
│   └── config.yaml              # Application configuration
│
├── 📁 data/
│   ├── processed/               # Processed article storage
│   └── raw/                     # Raw scraped links
│
├── 🖼️ images/                   # Downloaded article images
│
├── 📝 logs/                     # Application logs
│
├── 🔧 src/
│   ├── utils/                   # Core utility modules
│   └── websites/                # Scraping logic & main app
│
├── 🚀 main.py                   # Streamlit application entry point
├── 📋 requirements.txt          # Python dependencies
└── 🏃‍♂️ run_app.bat               # Windows launch script

git clone https://github.com/AdemCE-eng/Content_Inspiration.git
cd content_inspiration

# Create virtual environment
python -m venv venv

# Activate environment
# Windows:
venv\Scripts\activate
# macOS/Linux:
source venv/bin/activate

pip install -r requirements.txt

Create .env file in project root:

USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"

Note: Replace with your actual browser's user agent string. You can find this by searching "what is my user agent" in your browser.

# Install required model
ollama pull mistral

# Service starts automatically when needed

streamlit run main.py

run_app.bat

1. 🌐 Access Interface → Navigate to http://localhost:8501
2. ⚡ Run Pipeline → Execute 4-step scraping process via sidebar
   • Step 1 → Scrape article links from Google AI Blog
   • Step 2 → Download article content and metadata
   • Step 3 → Download and organize images
   • Step 4 → Generate AI summaries
3. 🔍 Explore Content → Use interactive features for content discovery

To use a different AI model, modify config/config.yaml:

ollama:
  model: "your-preferred-model"  # Change from default 'mistral'

• Check logs in logs/ directory
• Verify Ollama service status: ollama list
• Test user agent at: httpbin.org/user-agent

We welcome contributions! Here's how to get started:

# 1. Fork the repository on GitHub

# 2. Clone your fork
git clone https://github.com/YOUR-USERNAME/Content_Inspiration.git

# 3. Create feature branch
git checkout -b feature/amazing-feature

# 4. Make your changes and commit
git commit -m 'Add amazing feature'

# 5. Push to your fork
git push origin feature/amazing-feature

# 6. Create Pull Request