Luogu Advanced Competitive Programming Test (LACPT)

Introduction

Luogu Advanced Competitive Programming Test (LACPT) is a comprehensive benchmark suite designed to evaluate AI's coding abilities in high-difficulty algorithmic competitions. LACPT aims to serve as a rigorous benchmark for measuring AI's core capabilities in solving complex, non-standard programming problems, which are considered key components for achieving Artificial General Intelligence (AGI).

Authors

Authors: Shihao Ji, Zihui Song, Sumo Yu

Organizations: Luogu, Tengzhou No.1 High School

Project Structure

LACPT/ ├── 📁 src/ # Core source code
│ ├── 📁 judge/ # Judging module
│ │ ├── __init__.py
│ │ └── local_judge.py # Local judge
│ ├── 📁 prompts/ # Prompt templates
│ │ ├── __init__.py
│ │ └── competitive_programming.py
│ ├── 📁 generator/ # Test case generation
│ │ ├── __init__.py
│ │ └── test_case_generator.py # AI test case generator
│ ├── 📁 evaluator/ # Evaluation pipeline
│ │ ├── __init__.py
│ │ ├── evaluator.py # Main evaluator
│ │ └── model_interface.py # Model interface
│ └── __init__.py
├── 📁 data/ # Data directory
│ └── 📁 problems/ # Problem data
│ ├── 📁 a_plus_b/ # A+B problem
│ │ ├── problem.json # Problem description
│ │ └── test_cases.json # Test cases
│ └── 📁 fibonacci/ # Fibonacci problem
│ └── problem.json
├── 📁 scripts/ # Script tools
│ └── 📁 eval/
│ └── run_evaluation.py # Evaluation script
├── 📁 examples/ # Usage examples
│ └── quick_start.py # Quick start example
├── 📄 requirements.txt # Dependencies
└── 📄 README.md # Project documentation

Quick Start

1. Install dependencies

pip install -r requirements.txt

2. Set API key

export OPENAI_API_KEY="your_openai_api_key"

or

export ANTHROPIC_API_KEY="your_anthropic_api_key"

3. Run evaluation

python scripts/eval/run_evaluation.py --model openai --model-name gpt-4o

python scripts/eval/run_evaluation.py --model openai --problems a_plus_b fibonacci

python scripts/eval/run_evaluation.py --model openai --use-ai-generator

4. Quick example

python examples/quick_start.py

Problem Data Format

Each problem contains the following files:

problem.json

{
  "problem_id": "unique_id",
  "title": "Problem Title",
  "difficulty": "easy|medium|hard",
  "tags": ["tag1", "tag2"],
  "problem_statement": "Markdown format description",
  "input_file": "input.txt",
  "output_file": "output.txt",
  "time_limit": 1000,
  "memory_limit": 256,
  "reference_solution": {
    "language": "cpp|python",
    "code": "Reference code"
  }
}

test_cases.json (optional)

{
  "problem_id": "unique_id",
  "test_cases": [
    {
      "input": "test input",
      "expected_output": "expected output",
      "timeout": 5
    }
  ]
}

Capabilities Assessed

Mathematical Foundation and Theoretical Proof Ability:
Evaluates whether AI can effectively use mathematical tools to model algorithms and rigorously prove proposed solutions, including advanced concepts such as number theory, combinatorics, graph theory, and geometry.
Innovative Algorithm Design Ability:
Focuses on AI's ability to propose novel or highly optimized solutions for non-standard, open-ended problems, requiring creative thinking and unique algorithm design.
Efficient Code Implementation and Autonomous Error Correction:
Tests AI's ability to write high-performance code under strict time and memory constraints, and more importantly, to autonomously locate, diagnose, and fix errors, especially in the presence of complex corner cases.
Multi-level, Long-chain Logical Reasoning Ability:
Assesses whether AI can perform deep, multi-step logical reasoning and analysis, decompose complex problems, build logical chains, and switch/integrate between different abstraction levels.

Supported Models

OpenAI: GPT-4o, GPT-4, GPT-3.5-turbo
Anthropic: Claude-3-Opus, Claude-3-Sonnet, Claude-3-Haiku
Ollama: Local models (llama3, codellama, etc.)

Intended Use

AI model evaluation and benchmarking in advanced algorithmic competitions.
AGI capability research: advancing AI's abilities in mathematical reasoning, algorithm innovation, code implementation, and complex logical reasoning.
Defect analysis: helping researchers and developers identify weaknesses and limitations of current AI systems in complex programming tasks.

Limitations and Considerations

Domain specificity: LACPT focuses on algorithmic competitions; results may not generalize to other programming or software engineering tasks.
Code-centric: The test set mainly evaluates coding and algorithmic problem-solving, not broader software development skills.
Dynamic nature: Algorithmic problems and solutions evolve; LACPT will require regular updates to remain challenging and relevant.

Citation

@misc{luogu_llm_research_2025,
author = { Luogu LLM Research },
title = { LACPT },
year = 2025,
url = { https://huggingface.co/datasets/luogu-llm-research/LACPT },
publisher = { Hugging Face }
}