latest
This commit is contained in:
@@ -0,0 +1,212 @@
|
||||
# GAIS Data Import for ClassroomCopilot
|
||||
|
||||
This document describes the GAIS (Get All Information from Schools) data import functionality for ClassroomCopilot, which allows you to import publicly available school databases into the Neo4j database.
|
||||
|
||||
## Overview
|
||||
|
||||
The GAIS data import system is designed to import publicly available educational data from various sources, starting with Edubase All Data. The system follows Neo4j naming conventions and creates a comprehensive graph structure representing schools and their relationships.
|
||||
|
||||
## Neo4j Naming Conventions
|
||||
|
||||
The import system adheres to the following Neo4j naming conventions:
|
||||
|
||||
- **Node Labels**: PascalCase (e.g., `Establishment`, `LocalAuthority`, `EstablishmentType`)
|
||||
- **Relationships**: `HAS_` prefix followed by the target node label (e.g., `HAS_LOCAL_AUTHORITY`, `HAS_ESTABLISHMENT_TYPE`)
|
||||
- **Properties**: camelCase (e.g., `establishmentName`, `schoolCapacity`, `numberOfPupils`)
|
||||
|
||||
## Data Structure
|
||||
|
||||
### Main Nodes
|
||||
|
||||
1. **Establishment** - The primary school/educational institution node
|
||||
- Properties: URN, name, address, capacity, pupil counts, etc.
|
||||
- Unique identifier: `urn` property
|
||||
|
||||
2. **LocalAuthority** - Local authority governing the establishment
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_LOCAL_AUTHORITY`
|
||||
|
||||
3. **EstablishmentType** - Type of educational establishment
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_ESTABLISHMENT_TYPE`
|
||||
|
||||
4. **EstablishmentTypeGroup** - Group classification of establishment types
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_ESTABLISHMENT_TYPE_GROUP`
|
||||
|
||||
5. **PhaseOfEducation** - Educational phase (Primary, Secondary, etc.)
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_PHASE_OF_EDUCATION`
|
||||
|
||||
6. **GenderType** - Gender classification of the establishment
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_GENDER_TYPE`
|
||||
|
||||
7. **ReligiousCharacter** - Religious character of the establishment
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_RELIGIOUS_CHARACTER`
|
||||
|
||||
8. **Diocese** - Religious diocese (if applicable)
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_DIOCESE`
|
||||
|
||||
9. **GovernmentOfficeRegion** - Government office region
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_GOVERNMENT_OFFICE_REGION`
|
||||
|
||||
10. **DistrictAdministrative** - Administrative district
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_DISTRICT_ADMINISTRATIVE`
|
||||
|
||||
11. **MSOA** - Middle Super Output Area
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_MSOA`
|
||||
|
||||
12. **LSOA** - Lower Super Output Area
|
||||
- Properties: code, name
|
||||
- Relationship: `HAS_LSOA`
|
||||
|
||||
13. **Country** - Country of the establishment
|
||||
- Properties: name
|
||||
- Relationship: `HAS_COUNTRY`
|
||||
|
||||
## Usage
|
||||
|
||||
### Command Line
|
||||
|
||||
You can run the GAIS data import using the startup script:
|
||||
|
||||
```bash
|
||||
# Import GAIS data
|
||||
./start.sh gais-data
|
||||
|
||||
# Or directly with Python
|
||||
python main.py --mode gais-data
|
||||
```
|
||||
|
||||
### Programmatic Usage
|
||||
|
||||
```python
|
||||
from run.initialization.gais_data import import_gais_data
|
||||
|
||||
# Import the data
|
||||
result = import_gais_data()
|
||||
|
||||
if result["success"]:
|
||||
print(f"Successfully imported {result['total_rows']} records")
|
||||
print(f"Processing time: {result['processing_time']:.2f} seconds")
|
||||
print(f"Nodes created: {result['nodes_created']}")
|
||||
print(f"Relationships created: {result['relationships_created']}")
|
||||
else:
|
||||
print(f"Import failed: {result['message']}")
|
||||
```
|
||||
|
||||
## Data Sources
|
||||
|
||||
### Edubase All Data
|
||||
|
||||
The primary data source is the Edubase All Data CSV file, which contains information about all educational establishments in England and Wales.
|
||||
|
||||
**File Location**: `run/initialization/import/edubasealldata20250828.csv`
|
||||
|
||||
**Data Volume**: Approximately 51,900 records
|
||||
|
||||
**Key Fields**:
|
||||
- URN (Unique Reference Number)
|
||||
- Establishment details (name, type, status)
|
||||
- Geographic information (address, coordinates, administrative areas)
|
||||
- Educational characteristics (phase, gender, religious character)
|
||||
- Capacity and pupil numbers
|
||||
- Contact information
|
||||
- Inspection details
|
||||
|
||||
## Data Processing
|
||||
|
||||
### Batch Processing
|
||||
|
||||
The import system processes data in batches to optimize performance and memory usage:
|
||||
|
||||
- **Batch Size**: 1,000 records per batch
|
||||
- **Processing**: Nodes are created first, then relationships
|
||||
- **Error Handling**: Individual record failures don't stop the entire import
|
||||
|
||||
### Data Validation
|
||||
|
||||
The system automatically handles:
|
||||
- Empty/blank values (excluded from node properties)
|
||||
- "Not applicable" values (treated as empty)
|
||||
- Date format conversion (DD-MM-YYYY to ISO format)
|
||||
- Numeric value parsing
|
||||
- Duplicate node prevention
|
||||
|
||||
### Relationship Creation
|
||||
|
||||
Relationships are created using a two-pass approach:
|
||||
1. **First Pass**: Create all nodes and build a mapping of keys to node objects
|
||||
2. **Second Pass**: Create relationships between nodes using the mapping
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Memory Usage**: Data is processed in batches to minimize memory footprint
|
||||
- **Database Connections**: Uses connection pooling for efficient database access
|
||||
- **Duplicate Prevention**: Tracks created nodes to avoid duplicates
|
||||
- **Error Resilience**: Continues processing even if individual records fail
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
The GAIS import system is designed to be extensible for additional data sources:
|
||||
|
||||
1. **Governance Data** - School governance and management information
|
||||
2. **Links Data** - Relationships between schools and other entities
|
||||
3. **Groups Data** - Multi-academy trusts and federations
|
||||
4. **Additional Sources** - Other publicly available educational datasets
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **File Not Found**: Ensure the Edubase CSV file is in the correct location
|
||||
2. **Database Connection**: Verify Neo4j is running and accessible
|
||||
3. **Memory Issues**: Reduce batch size if processing large datasets
|
||||
4. **Permission Errors**: Check file permissions for the CSV data file
|
||||
|
||||
### Logging
|
||||
|
||||
The system provides comprehensive logging:
|
||||
- Import progress updates
|
||||
- Error details for failed records
|
||||
- Performance metrics
|
||||
- Node and relationship creation counts
|
||||
|
||||
### Testing
|
||||
|
||||
Use the test script to verify functionality:
|
||||
|
||||
```bash
|
||||
python test_gais_import.py
|
||||
```
|
||||
|
||||
## Data Quality
|
||||
|
||||
The import system maintains data quality by:
|
||||
- Filtering out invalid or empty values
|
||||
- Converting data types appropriately
|
||||
- Maintaining referential integrity
|
||||
- Providing detailed error reporting
|
||||
|
||||
## Schema Compatibility
|
||||
|
||||
The imported data is compatible with the existing ClassroomCopilot schema and can be integrated with:
|
||||
- Calendar structures
|
||||
- User management systems
|
||||
- Educational content management
|
||||
- Analytics and reporting tools
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions related to the GAIS data import:
|
||||
|
||||
1. Check the logs for detailed error information
|
||||
2. Verify data file format and content
|
||||
3. Ensure database connectivity and permissions
|
||||
4. Review the Neo4j schema constraints and indexes
|
||||
Reference in New Issue
Block a user