Participants in this study consisted of 242 ninth-grade students from three public junior high schools in Bengkulu, Indonesia, during the 2024/2025 academic year. The selected schools were: SMP Negeri 19 (three classes), SMP Negeri 21 (three classes), and SMP Negeri 3 (one class). These schools were intentionally selected using purposive sampling to represent a range of school performance levels categorized as low, medium, and high based on the schools' average scores in summative science assessments over the past two years (2022)(2023)(2024).

In addition to academic level, the sample considered diversity in gender, socio- economic background, and students’ prior environmental understanding, which was measured using a short diagnostic questionnaire administered before the main instrument. The average age of participants was 14 years, with a relatively balanced distribution between male and female students.

The empirical validation of the AABAI instrument was conducted through a one-time testing design (cross-sectional) without pre- test or post-test. The focus of this validation phase was to examine the instrument’s item validity, reliability, and response patterns using Rasch modeling, not to measure learning gains over time. The choice of coastal schools was intentional, given that Bengkulu is located along the western coastline of Sumatra, where coastal community culture remains strong. Embedding coastal environmental issues in the instrument aims to create a more contextual and meaningful learning experience for students.

The instrument, designed to serve as an indicator of environmental literacy, underwent content validation through expert evaluation. A panel of six assessors participated in the validation process, comprising two specialists in educational assessment, two experts in science education, and two field practitioners. In addition, three media experts evaluated the AABAI instrument. To assess its readability, the instrument was administered to a sample of 242 students. Subsequently, item trials were conducted to gather empirical evidence supporting the instrument's quality.

The complete AABAI development pro- cedure is shown in Figure 1.

The formative application used in this study is an AI-based automatic assessment platform called Formative. This platform was selected for its real-time feedback capabilities, the ability to monitor student learning dynamically, and its varied question formats, including multimedia integration. These features make it particularly suitable for embedding contextualized science problems related to coastal environmental issues. The menus available for displaying questions are also very varied. It can display images, videos, graphics, sound, and simula- tions so that students avoid answering ques- tions carelessly. Question types also vary: multiple choice, complex multiple-choice, case studies, short descriptions, long descrip- tions, answering with audio, uploading ima- ges, and many more. This variation in the appearance of questions will minimize stu- dents answering carelessly.

However, the implementation of AIbased tools in education raises concerns regarding algorithmic bias, particularly when applied to culturally diverse student populations (Blodgett et al., 2020)(Ferrara, 2024). To address this, several mitigation strategies were applied. First, the content used in the assessment was localized to reflect the cultural and environmental context of coastal communities in Bengkulu, including traditional marine practices and local environmental challenges. This adaptation ensures that language, examples, and values embedded in the tasks are relevant and inclusive.

Second, while the AI features of Formative facilitate automatic scoring and feedback, teachers retained control over final interpretations, especially in responses that may involve cultural reasoning, local dialects, or non-standard expressions that AI might misinterpret. This hybrid model combining AI automation with teacher judment reduces the risk of misclassification and cultural insensitivity (Williamson & Eynon, 2020) .

This approach aligns with the principles of AI fairness in education, which emphasize the need for transparency, contextual relevance, and human oversight to prevent reinforcing systemic inequalities (Luckin et al., 2019). By integrating local context and maintaining teacher involvement, the use of Formative in this study supports equitable and culturally responsive assessment practices.

After the instrument is ready, the validity and reliability of the test questions are tested. The content validity test is given to material and instrument experts to assess suitability, completeness, and readability. Media validity is given to media experts to determine the quality of AI and functionality, display design, usefulness in learning.

The V Aiken formula determines the va- lidity of the test instrument’s content. The content validity coefficient is based on expert assessments. The V Aiken index value is calculated as follows:

Where s is the assessment score minus the lowest score on the scale, n is the number of experts, and c is the number of categories on the assessment scale. The V value is between 0 and 1, where the closer it is to 1, the higher the level of content validity. (Aiken, 1985); Azwar, 2012). According to(Azwar, 2015), the validation criteria for the AABAI instrument in assessing environmen- tal literacy are classified into five distinct categories, as shown in Table 1.

The validation of the AABAI instrument was further supported by empirical data ob- tained through the analysis of test item res- ponses using polytomous data. This data was examined using Item Response Theory (IRT), specifically employing the Rasch mo- del or the Partial Credit Model (PCM) based on the one-parameter logistic (1-PL) appro- ach. The analysis was conducted with the assistance of Quest and Microsoft Excel sof- tware. Quest was utilized to assess item fit statistics, test reliability, and item difficulty indices. The Excel program was used to dis- play information on student ability profiles. According to(Hambleton & Swaminathan, 1985), test items are considered to align well with the Partial Credit Model (PCM) when the INFIT Mean Square (MNSQ) va- lues fall within the range of 0.5 to 1.5, and the INFIT t-statistics lie between -2.0 and 2.0. The reliability of the integrated asses- sment instrument can be calculated using the Quest software. The reliability index is ob- tained from the output file with the .sh exten- sion, specifically found in the Item Estima- tion Summary section. The higher the relia- bility coefficient, the more reliable the instrument and the smaller the possibility of error (Subali, 2015) .(George & Mallery, 2020) categorize the reliability coefficients as shown in Table 2.

Student feedback on the readability of the instrument, as product users, was collected through a questionnaire that had been previously validated by expert reviewers.

The resulting scores were then translated into feasibility categories based on the crite- ria outlined in Table 3. (Sumadi et al., 2015) .

The level of difficulty of the environ- examining the estimate (b) values, as mental literacy test items was determined by presented in Table 4(Bond & Fox, 2015) .

Meanwhile, the media validity coefficient is based on expert assessments. While the readability of the text of the reference score change instrument is in Table 5 (Mardapi, 2008).

The content validity of the test instru- ment developed in this study was evaluated by six expert assessors using a four-point rating scale. According to Aiken’s criteria, the minimum acceptable value for the Aiken's V coefficient in this context was 0.78 at a 0.05 level of significance (Aiken, 1985). The results of this validation analysis are illustrated in Figure 2.

The analysis results show that the esti- mated item reliability is in the range of 0.81 to 1.23, which means that the sample is con- sistent with the items being evaluated and is very good, as seen in Table 6. The results of the goodness-of-fit analysis, based on the INFIT Mean Square (MNSQ) parameters, demonstrate that the AABAI instrument used to assess environmental literacy satisfies the statistical fit requirements, as detailed in Table 6.

Figure 3 presents a set of 20 items analyzed using the Rasch model. According to (Hambleton & Swaminathan, 1985), an item is considered to have acceptable quality if its difficulty index falls within the range of -2 to +2 logits, indicating an appropriate le- vel of difficulty.

The analysis presented in Figure 3 indi- cates that all 20 items are classified within the acceptable or "good" category based on their difficulty levels.

An effective test item is one that satisfi- es the criteria for both validity and reliability, and possesses an appropriate level of diffi-

culty. Table 7 presents the results of the analysis conducted to determine the difficul- ty levels of the test items.

Table 7 shows that 16 questions (80%) fall into the moderate category, 2 questions (10%) fall into the difficult category, and 2 (10%) questions fall into the easy category.

The results of the media validation test (AI quality and functionality, display design (user interface), usefulness in learning) in-

volving three media validators. Figure 4 shows the results of the media validation on the criteria of “very good” without revision. Therefore, this test instrument is suitable for use.

The purpose of the readability test is to assess the complexity level of the written text. This evaluation provides an estimate of readers' comprehension at the sentence level. Readability, which reflects how easily or difficultly the content of a text can be un- derstood, is illustrated in Figure 5.

The results of the instrument readability test (seen from the aspects of clarity of language and sentence structure, suitability of content to student ability level, and clarity of intent and interest in questions) involving 242 students showed “very good” results with a slight revision of the simplification of terms. After being revised, this test instrument is suitable for use.

Based on the environmental literacy test results, most students are in the moderate category (95%), showing a good basic understanding of environmental issues such as waste and global warming. Meanwhile, 5% showed high environmental literacy scores.

The AABAI empirical test was conduc- ted to determine the suitability of environ- mental literacy indicators with the formula- ted questions. This test instrument focuses on environmental issues as the primary material, which is then divided into sub-topics such as food availability, environmental health, energy crisis, and global warming. This topic is still limited but promising, especially in the context of strengthening students’ aware- ness and competence towards environmental issues (Syahmani et al., 2021) . This study uses environmental literacy indicators based on(Santosa et al., 2021), then integrated with local cultural traditions of coastal communi- ties so that environmental issues are more contextual with a focus on coastal areas and adapted to the cognitive abilities of C4-C6.

The content validity of the developed test instrument was reviewed by six expert validators using a four-point assessment scale. This validation step was conducted prior to empirical testing. Following (Aiken, 1985) criteria, a minimum acceptable Aiken’s V coefficient of 0.78 at the 0.05 significance level was adopted. As presented in Figure 2, all item scores ranged from 0.98 to 1.05, indicating that each item significantly exceeded the threshold and thus fulfilled the content validity criteria. Based on the classification in Table 1, all items fall into the “very good” category.

These findings are consistent with previous studies, such as (García-Ceberino et al., 2020), which stated that items with an Aiken’s V value ≥ 0.77 are considered valid, and (Mistiani et al., 2022), who emphasized that instruments with V values above 0.80 indicate high content validity.

However, such uniformly high values across all items raise important questions about the underlying context. Several factors may have contributed to this outcome. First, the items were developed based on science concepts aligned with the existing junior high school science curriculum, especially on topics like waste, pollution, and coastal ecosystems issues that are not only taught in schools but also highly visible in students’ daily lives in Bengkulu’s coastal environment. This contextual familiarity may have led to clearer item wording and stronger agreement among validators regarding content relevance.

Second, during the development process, items were designed in collaboration with local educators and environmental experts, incorporating authentic and culturally relevant environmental issues (e.g., marine debris, coral reef damage, and traditional coastal practices). This alignment between local context and scientific content likely made the items easier to evaluate positively, both in terms of curriculum relevance and construct representation.

Third, the validators were selected based on expertise in science education, assessment, and environmental literacy, which may have introduced consistency in judgment criteria, leading to less variation in scoring.

While these results are encouraging, they may also suggest that future iterations of the instrument should include more complex or unfamiliar scenarios to assess deeper conceptual understanding or critical thinking, particularly for higher-order cognitive demands. Including items that go beyond recall and recognition could provide a more nuanced picture of environmental literacy levels and challenge students to engage in reflective decision-making.

The high content validity (Aiken’s V values ranging from 0.98 to 1.05) and strong empirical reliability of the AABAI instrument are not only statistically significant but also carry important practical implications for implementation in real educational settings. For teachers, the validated and reliable nature of the instrument enhances confidence in using AABAI as a dependable tool for assessing students’ environmental literacy. When teachers are assured that each item has been reviewed for clarity, relevance, and representativeness, they are more likely to integrate the assessment into classroom practice without concerns about ambiguity or misalignment with curriculum goals.

Moreover, the reliability indicators both item and person reliability demonstrate that the instrument performs consistently across diverse student groups. This consistency is crucial when used in schools of varying academic levels (low–medium–high) as it ensures measurement fairness and comparability of results across contexts.

Importantly, the instrument was deliberately developed to reflect local environmental and cultural contexts, especially those specific to coastal communities in Bengkulu. Items included themes such as marine debris, traditional fishing practices, coral reef degradation, and community-based coastal conservation rituals. This contextual integration helps ensure that AABAI is not only scientifically accurate but also culturally resonant, allowing it to capture local environmental wisdom that may not be visible in standardized national assessments.

As a result, the instrument’s high validity and reliability enable it to function not just as an academic assessment tool, but also as a medium for cultural affirmation helping students recognize the value of their community’s environmental knowledge and practices. This could also serve as a stimulus for place-based learning, where science education becomes more meaningful because it is rooted in students’ lived experiences.

The results of the test of the level of difficulty of the test questions are shown in Table 5. A total of 16 questions (80%) are in the moderate category, two questions (10%) are in the difficult category, and 2 (10%) questions are in the easy category by looking at the estimated value (b). The distribution of the questions’ difficulty level has met the provisions for the distribution of the difficulty level. Data collection was carried out by providing 20 AABAI questions in 60 minutes. When creating the questions, the questions were classified into certain difficulty levels (easy, moderate, and difficult) based on the expected answer requests in AABAI. Questions in the form of projects, which require answers to design/draw/plan a project, are considered the most difficult. Meanwhile, questions with answer requests, such as complex multiple-choice, which only choose more than one correct answer, are considered the easiest questions. A test item is considered to be of good quality when its level of difficulty is balanced neither excessively difficult nor overly simple meaning it falls within the moderate or acceptable range. A test is considered good if it is reliable. As presented in Table 2, the test reliability falls within the "good" category, indicating that the instrument is consistently dependable for assessing students' environmental literacy.

Figure 4 shows that in terms of AI quali- ty and functionality, an average of 3.89 is obtained, which is included in the “very good” category. Regarding display design (user interface), an average of 3.78 is ob- tained, which is included in the “very good” category. Regarding its effectiveness in the learning process, the AABAI instrument achieved an average score of 3.33, which falls within the "very good" category. This outcome suggests that AABAI fulfills the criteria of a high-quality assessment tool, particularly in terms of artificial intelligence performance, system functionality, user in- terface accessibility, and its overall contribu- tion to enhancing learning experiences. This indicates that AABAI is worthy of being implemented in the educational context as an assessment innovation that is efficient, adap- tive, and responsive to the needs of students. Empirical support from international litera- ture further strengthens these findings. (Cui et al., 2018) intelligent interface design in an AI-based assessment system increased stu- dent engagement by 25% in a field study involving over 20.000 participants. A study of the adaptive learning system “Yixue” in China reported significant improvements in academic performance compared to traditio- nal methods. (Awang et al., 2024) AI-based adaptive learning platforms increase user motivation, conceptual understanding, and satisfaction. In addition, adaptive AI systems can personalize the learning experience and improve student engagement and performan- ce (Gligorea et al., 2023) . The combination of these results shows that AABAI is techni- cally superior and has been proven to contri- bute positively to the effectiveness of lear- ning and student engagement.

Figure 5 shows that the clarity of lan- guage and sentence structure, suitability of content to student ability level, and intent and interest in questions are in the “very go- od” category. This means that the AABAI assessment instrument has a "very good" level of readability in terms of clarity of lan- guage and sentence structure, suitability of content to student ability level, and clarity of purpose and interest in questions. So AABAI is considered effective in conveying the pur- pose of the evaluation communicatively and easily understood by students and has the potential to increase their involvement in the assessment process.(Yaacoub & Prevost, 2025) highlighted that language-based AI feedback mechanisms significantly improved the readability and adaptability of questions, resulting in more meaningful student respon- ses. Furthermore, (Ali et al., 2025) was found that using AI-based assessment tools desig- ned with a focus on readability and language structure effectively strengthened students’ positive perceptions of the instruments’ vali- dity and relevance.

Figure 6 shows that the majority of students (95%) are categorized as having moderate environmental literacy, indicating that they possess a basic understanding of environmental issues such as waste, global warming, and their ecological and social impacts. This suggests that students are generally aware of key environmental challenges, but have not yet developed the higher-level skills necessary for critical evaluation or transformative action.

One likely reason is the limited contextualization of previous science learning approaches. Based on curriculum observations and teacher interviews, environmental topics such as pollution, climate change, and coastal degradation were often delivered through lecture-based methods, lacking direct connection to students' local environments or cultural experiences. This may have hindered students from transferring conceptual understanding into real-world applications, which is essential for reaching the high literacy level.

Furthermore, students limited exposure to higher-order assessment formats, such as project-based or problem-based learning that require critical thinking, collaborative inquiry, and the ability to connect scientific knowledge with sustainable action. Since most classroom assessments focused on recall and factual understanding, students were less prepared to handle the complex, scenario-based questions embedded in the AABAI instrument.

Only 5% of students fall into the high category, which reflects their ability to not only comprehend environmental problems but also to apply that knowledge in analyzing real-world issues and proposing sustainable solutions. This profile aligns with findings by (Örs, 2022), who reported that even students with strong environmental attitudes may lack depth in conceptual understanding. Similar trends were reported in the United States by the NOAA's National Environmental Literacy Assessment (NELA), which revealed that most students scored in the moderate range and emphasized the need for context-rich and cognitively challenging instruction (McBeth et al., 2011).

To explore possible factors influencing these results, correlation analysis was conducted between item difficulty levels (as measured using Rasch logit scores) and student performance levels. The results show a positive correlation (r = 0.41) between item difficulty and student achievement category, suggesting that students in the high category were more likely to succeed on high-difficulty (higher-logit) items, while those in the moderate category performed well only on low- to medium-difficulty items. This indicates that students’ conceptual depth is still limited, and higher-order items remain challenging for the majority.

Furthermore, an analysis across the three sampled schools (classified as low, medium, and high based on academic performance) reveals differences in literacy profiles. In the high-performing school, 12% of students achieved high literacy, compared to just 3% in the medium-performing school, and none in the low-performing school. This suggests that school context and academic culture may play a role in shaping environmental literacy outcomes. Students in more academically advanced schools are likely more exposed to enriched science learning environments, better instructional strategies, and more frequent engagement with environmental themes in both formal and informal learning contexts.

These findings imply that environmental literacy, while generally present at a basic level among students, remains unevenly developed and sensitive to contextual factors such as school quality and instructional approach. To move students beyond the moderate level, there is a need for differentiated, context-specific, and higher-order learning strategies that engage students in problem-solving, action-oriented projects, and critical reflection on real environmental issues in their communities.

While the technical success of AABAI, demonstrated through high validity, reliability, and user adaptability is a crucial foundation, its transformative value lies beyond technical functionality. AABAI is not only designed to measure environmental literacy, but also to stimulate reflection, build awareness, and encourage pro-environmental behaviors among students, especially those in coastal communities vulnerable to ecological degradation.

Through its scenario-based questions, AABAI presents students with authentic environmental challenges that are closely tied to their daily lives, such as plastic pollution on beaches, coral reef damage, and mangrove conservation. These scenarios require students not only to recall facts, but also to reflect on the causes, consequences, and solutions a process that activates higherorder thinking and personal responsibility. This is aligned with the constructivist principle that meaningful learning occurs when students can relate new information to their lived experiences (Piaget, 1972)(Vygotsky, 1978).

Moreover, the immediate and personalized feedback generated by AABAI helps students understand the implications of their choices, correct misconceptions, and explore sustainable alternatives. For example, if a student selects an ineffective or unsustainable solution in a coastal scenario, the system explains why it is problematic and suggests community-based alternatives thus reinforcing both cognitive accuracy and behavioral guidance.

Therefore, AABAI plays a dual role: as an assessment tool that measures environmental literacy dimensions (knowledge, attitude, behavior, reflection) and as a learning tool that fosters environmental awareness and motivates students to act. Future studies should include longitudinal designs to better capture AABAI's influence on long-term behavior change and its potential as a catalyst for community-based environmental education.

The novelty of this research lies in the development of AABAI an AI-based asses- sment tool that uniquely integrates adaptive feedback, environmental literacy dimensi- ons, and local coastal cultural context. Unli- ke previous studies such as (Cui et al., 2018), which focused on general AI-assisted for- mative assessments, or (Awang et al., 2024) and (Gligorea et al., 2023), which emphasi- zed standardized instruments with limited cultural depth, AABAI embeds authentic, place-based scenarios rooted in the lived experiences of Bengkulu’s coastal commu- nities. These include traditional ecological practices, local environmental challenges, and cultural values related to sustainability. Furthermore, AABAI not only automates scoring but also provides meaningful, real time feedback tailored to students' respon- ses, encouraging critical reflection and pro- environmental behavior. This combination of automated assessment, cultural contex- tualization, and pedagogical impact repre- sents a significant advancement over exis- ting instruments, positioning AABAI as a transformative tool that not only measures but also fosters environmental awareness and local action an innovation not yet seen in prior AI-driven environmental literacy research.