This study employed a design research methodology of the development studies (²; ³¹). The developmental process followed Tessmer’s formative evaluation model ⁴¹, consisting of a preliminary phase, formative evaluation (self-evaluation, expert review, one-to-one, small group, and field test), and an assessment phase ⁴⁹. This approach was selected to produce PISA-like task in the Uncertainty and Data domain that are valid, practical, and potentially effective for enhancing statistical literacy.

The participants were Grade IX students from a state junior high school in Jambi, Indonesia. Student categorization into low, medium, and high achievement levels was determined based on their mathematics daily test scores, as these scores reflect students’ recent mastery of statistical and data-related content. Three students representing each achievement category participated in the one-to-one evaluation stage, while nine students took part in the small-group evaluation. There were 37 students who involved in the field test stage to examine the practicality and potential effects of the developed tasks. In addition, one mathematics teacher provided curricular and classroom-based judgments, while three mathematics education researchers specializing in PISA item development and statistical literacy served as expert validators to evaluate the content validity and construct quality of the tasks.

Figure 1 presents the overall research procedure adapted from ⁴¹ and ⁴⁹ formative evaluation model. The diagram illustrates the sequential flow of activities across the preliminary phase, formative evaluation, and assessment phase. As shown in Figure 1, the procedure consisted of three phases. The preliminary phase involved document analysis of the PISA 2022 framework, the Indonesian Merdeka Curriculum (Phase D), relevant PISA items, and prior studies to construct Draft 1 containing item specifications, scoring rubrics, and five climate change contextual themes global warming, deforestation, natural disaster frequency, air temperature anomalies, and Jakarta land subsidence selected for their data richness and alignment with students' lived experiences and the SDGs ¹⁷. The formative evaluation stage refined the tasks through iterative cycles: self-evaluation produced Prototype 1; expert review generated Prototype 2; one-to-one trials improved readability; small-group trials assessed practicality and led to Prototype 3; and a full-class field test produced the Final Prototype. The assessment phase examined the potential effectiveness of the final tasks by analyzing students' reasoning and the appearance of statistical literacy indicators.

Data were collected through six techniques applied across the development stages. First, document analysis of the PISA 2022 framework, Merdeka Curriculum standards, and prior studies was conducted during the preliminary phase to inform the construction of the initial task draft. Second, walkthroughs were used during the expert review stage, in which validators provided written feedback and recommendations on content, construct, and language quality. Third, validity

questionnaires were administered to the four validators during the expert review stage to obtain quantitative ratings across nine indicators covering content, construct, and language. Fourth, student practicality questionnaires were administered during the small-group and field test stages to assess usability, readability, and feasibility. Fifth, written task responses from 37 field-test students were collected to examine the potential effects of the final set of ten PISA-like tasks on students' statistical literacy. Sixth, semi-structured interviews were conducted with selected students at each evaluation stage at one-to-one to probe comprehension and language clarity, at small group to assess usability, and at field test to explore how the climate change context supported students' data reasoning.

Data analysis combined qualitative and quantitative techniques at different stages. Qualitatively, walkthrough and interview data were analyzed descriptively to identify issues of content, construct, and linguistic clarity, and observational notes capturing student behaviors such as expressions of confusion, problem-solving strategies, and engagement with data representations were analyzed at each stage to inform task revision.

The quantitative data analysis procedures included the evaluation of content validity, practicality, and the potential effect of the developed PISA-like tasks. The scoring procedures, assessment criteria, and corresponding references used in this study are summarized in Table 1. As shown in Table 1, the validity and practicality analyses were conducted using questionnaire data scored on a five-point Likert scale and converted into percentages of the maximum possible score (³⁵; ⁴⁴). Meanwhile, the potential effect analysis was based on students’ written responses using a partial-credit scoring rubric adapted from ²⁹. The resulting scores were categorized into high, moderate, and low levels to determine the extent to which the developed tasks supported students’ statistical literacy performance.

To further examine students’ statistical literacy performance, students’ responses were analyzed based on three statistical literacy indicators adapted from ³⁶ and ⁹. The indicators and their operational explanations are presented in Table 2. The indicators presented in Table 2 served as analytical categories for identifying patterns of students’ statistical reasoning when solving the climate change-based PISA-like tasks. Through these indicators, the analysis focused on how students understood the problem context, processed the provided data, and interpreted the results to produce evidence-based conclusions.

The preliminary phase aimed to establish a coherent foundation for task development by aligning student characteristics, curriculum demands, PISA framework requirements, and the selected climate-change contexts. This phase consisted of preparatory analyses followed by the construction of an initial task draft.

The process began with a student analysis conducted in collaboration with mathematics teachers at Secondary School 6 Kota Jambi. The analysis focused on students’ academic diversity and learning readiness to ensure that participants selected for each evaluation stage appropriately represented varying ability levels. Based on this analysis, three Grade IX C students were selected for the one-to-one stage, nine Grade IX B students participated in the small-group evaluation, and one intact Grade IX A class was designated for the field test.

In terms of curriculum, the secondary school implements the Kurikulum Merdeka (Independent Curriculum–IC), which is consistent with the curriculum framework adopted by secondary schools across Jambi Province, including schools participating in the development of PISA-like tasks within the Uncertainty and Data domain. The IC places substantial emphasis on data literacy and evidence-based reasoning, particularly through competencies requiring students to analyze, evaluate, and interpret quantitative information in contextual situations. These competencies conceptually align with the construct of statistical literacy, which encompasses the ability to interpret data representations, identify patterns and variability, evaluate quantitative claims, and formulate evidence-based conclusions in situations involving uncertainty.

Accordingly, the learning objectives and task indicators developed in this study were aligned with the learning outcomes specified at the end of Phase D of the IC, particularly within the Data Analysis and Probability element, as presented in Table 3. The Data Analysis and Probability learning outcomes provide a strong conceptual and curricular basis for the development of statistical literacy-oriented PISA-like tasks. The emphasis on interpreting data, identifying trends, and generating predictions is particularly relevant to the Uncertainty and Data domain, as these competencies

require students to engage in statistical reasoning beyond procedural computation. Consequently, the curriculum supports the integration of authentic and data-intensive contexts, such as climate change, to facilitate students’ capacity to critically interpret information and make informed decisions based on empirical evidence.

To ensure alignment with international assessment standards, the PISA framework was analyzed with particular attention to content, context, and mathematical processes. The content domain emphasized Uncertainty and Data, while the contexts were drawn from scientific and social issues related to climate change. In terms of mathematical processes, the tasks were designed to engage students in the formulate–employ–interpret (FEI) cycle as defined in the PISA framework ²⁹.

As part of this analysis, several released PISA items were examined to understand their structure, use of data representations, and cognitive demands. One example is the Charts task from PISA 2012, which presents a bar chart showing monthly CD sales of several music bands and asks students to interpret trends across time (see Figure 2). This task requires students to identify relevant variables, compare values across categories, and reason about changes over successive periods rather than simply performing routine calculations.

Figure 2 The Charts task was not reused in this study; instead, it served as a conceptual reference for designing PISA-like task that emphasize trend analysis and interpretation of data over time. Drawing on this structure, the developed tasks in this study were designed to require students to analyze historical data and make predictions about future conditions. For example, students were asked to estimate future temperature values based on trends observed in climate data from previous years.

This design choice intentionally moved beyond routine procedural tasks and encouraged multi-step reasoning and contextual interpretation. Although a formal psychometric level classification was not conducted at this stage, the tasks were constructed to reflect the cognitive characteristics of PISA items that involve reasoning with trends and making data-based predictions. This consideration informed the intended level of task complexity without imposing rigid level claims.

The final in the preliminary phase was the construction of the initial task draft. A test specification table (blueprint) was developed to map curriculum learning outcomes, PISA framework components, statistical literacy indicators, and climate-change contexts. Based on this blueprint, a set of open-ended, constructed-response tasks was written, accompanied by a scoring rubric designed to support consistent and objective assessment of students’ responses. The output of this phase was the initial version of the developed instrument, referred to as Prototype I, which served as the basis for subsequent refinement during the formative evaluation phase.

The formative evaluation stage was conducted through a formative evaluation cycle aimed at refining and validating the developed PISA-like tasks. This stage consisted of several iterative stages, namely self-evaluation, expert review, one-to-one evaluation, small-group evaluation, and field testing. Each stage played a specific role in improving the quality of the tasks in terms of content validity, clarity, contextual relevance, and their ability to elicit students’ statistical literacy within the Uncertainty and Data domain.

Self-evaluation

At the self-evaluation stage, the researcher conducted an internal review of Prototype I by referring to the PISA framework, particularly the Uncertainty and Data domain and the FEI mathematical process ²⁹. This stage aimed to ensure that the task was conceptually aligned with statistical literacy indicators and emphasized reasoning with data and uncertainty rather than routine procedural calculations.

The task was developed using an informational poster presenting projected data on the percentage of Jakarta’s area potentially inundated by seawater due to continued land subsidence. The “Jakarta Tenggelam” context was selected as an authentic climate change is arelated issue involving uncertainty, trend analysis, and long-term data interpretation. Students were asked to estimate the year in which 50% of Jakarta’s area might be submerged based on the observed data trend and to justify their prediction using evidence from the poster.

During self-evaluation, the researcher examined whether the task required students to identify relevant quantitative information, recognize trends in the projected data, and extrapolate these trends to make a reasoned prediction. Particular attention was given to engaging the interpret phase of the PISA mathematical process, with the task intentionally designed to avoid direct formula application and instead emphasize data interpretation and justification under uncertainty.

Based on this evaluation, several revisions were made, including refining the wording for clarity, strengthening the conceptual link between land subsidence and climate change, and ensuring that the prompt explicitly directed students to base their predictions on data trends rather than subjective opinion. Visual elements were also reviewed to support clearer comparison across years.

Figure 3 illustrates the revised Prototype I following the self-evaluation stage, highlighting the improved clarity of the problem statement and the visual presentation of projected inundation data that supports students’ trend analysis and prediction. Figure 3 presents Prototype I of the developed PISA-like task, which uses the "Jakarta is sinking?!?" context as a representative example of the climate change theme. This task presents an informational poster containing projected data on the percentage of Jakarta's area potentially inundated by seawater due to continued land subsidence, a phenomenon driven by a combination of excessive groundwater extraction, increased load from tall buildings, and rising sea levels resulting from climate change. Students are asked to estimate the year in which 50% of Jakarta's area will be flooded and to justify their prediction mathematically based on the observed data trend. This context was selected because it inherently involves uncertainty

trend analysis, and long-term data interpretation, which are central characteristics of the Uncertainty and Data domain, while also representing a locally and globally relevant climate issue that is meaningful to Indonesian students.

Expert review

The expert review stage was conducted to obtain systematic feedback and recommendations on the quality of Prototype I, with particular emphasis on content validity, construct validity, and language clarity. This stage aimed to ensure that the developed PISA-like task met essential quality standards before proceeding to subsequent formative evaluation phases. A summary of the experts’ quantitative and qualitative evaluations is presented in Table 2 and Table 3.

Four experts participated in this stage, consisting of three university lecturers in mathematics education and one experienced junior high school mathematics teacher. The lecturers have extensive experience in mathematics education research and PISA-oriented task development, including the design and evaluation of PISA-like assessment items, while the teacher has more than 15 years of classroom teaching experience and has been involved in school-based assessment practices. This combination of academic and practical expertise ensured a comprehensive evaluation of the task from both theoretical and classroom perspectives.

The quantitative results of the expert validation are summarized in Table 4. The validation was conducted using a five-point Likert scale (1 = strongly disagree to 5 = strongly agree) across nine indicators covering content, construct, and language aspects.  The validation results indicate that Prototype I achieved an average validity percentage of 83.9%, which falls within the very valid category ⁴⁴. This result demonstrates that the developed tasks were considered

appropriate by the validators across content, construct, and language dimensions. From the content perspective, the experts agreed that the tasks were consistent with the PISA 2022 Uncertainty and Data framework and successfully integrated statistical literacy indicators, particularly problem understanding, data processing, and data interpretation, through authentic climate change situation. The validators highlighted that the use of temperature trends, emission data, and environmental statistics provided meaningful opportunities for students to engage in evidence-based reasoning and interpretation of uncertainty-related information. In terms of contruct validity, the validators considered the organization of the task components, including the stimulus presentation, question sequencing, and scoring rubric, to be coherent and capable of eliciting the intended statistical reasoning processes. The contextual stimuli were judged to sufficiently support students in identifying relevant information, analyzing quantitative relationships, and constructing data-based conclusions. Nevertheless, several revisions were recommended, particularly regarding the refinement of graphical representations and the simplification of several question prompts to improve clarity and reduce potential misinterpretation. From the language perspective, the validators agreed that the language used in the tasks was generally appropriate for junior secondary school students and aligned with students’ cognitive characteristics. Minor revisions were suggested to improve sentence effectiveness, consistency of terminology, and readability of instructions. These revisions were subsequently incorporated into Prototype II, resulting in tasks that were not only theoretically aligned with the statistical literacy framework but also practically suitable for classroom implementation.

In addition to numerical ratings, the experts provided written comments and suggestions that offered deeper insight into specific aspects requiring refinement. These qualitative findings are summarized in Table 5.

Regarding language clarity, experts recommended improving sentence formulation to enhance student comprehension, including correcting punctuation usage, such as limiting the question mark in the item title “Jakarta is singking ?!?” and paraphrasing terms like “trend,” which were considered potentially difficult for students at the junior secondary level. From a construct perspective, experts suggested adding clear titles or captions to images and tables (e.g., “Figure 1. …”) to support accurate interpretation of visual representations. Furthermore, they advised avoiding bold formatting in question statements to ensure that students rely on data interpretation and visual literacy rather than textual emphasis.

Overall, although Prototype I was judged to be very valid, the experts’ suggestions provided valuable guidance for refinement. These recommendations were used as the basis for revising Prototype I to improve clarity, readability, and alignment with students’ mathematical literacy skills, particularly in interpreting data and representations within a climate change context.

One-to-one

The one-to-one evaluation was conducted to identify students’ difficulties in understanding the task, particularly in relation to language use and data interpretation. Overall, students were able to recognize the climate change issue embedded in the task and attempted to make predictions based on the data provided. This finding suggests that the task was generally accessible and capable of eliciting students’ statistical reasoning within the Uncertainty and Data domain.

Table 6 presents students’ comments and responses during the one-to-one evaluation. While one student reported no difficulties, two students indicated problems in understanding the term “trend.” These responses highlight that, although students could interpret the visual data and observe numerical changes, specific terminology posed challenges that affected their reasoning process.

Further qualitative analysis revealed that the main difficulty was linguistic rather than mathematical. In particular, unfamiliarity with the term “trend” limited students’ ability to justify their predictions mathematically. Students also noted that clearer visual labels and more explicit question wording would better support their reasoning. This issue is illustrated in the following interaction between the researcher I and a student (S):

This interaction shows that the students’ difficulty did not stem from interpreting numerical data, but from understanding specific terminology that guides statistical reasoning; therefore, based on the results of the expert review and one-to-one evaluation, several revisions were implemented, including simplifying the introductory paragraph to help students focus on the data, adding clear labels to visual representations, revising visual information to ensure accurate geographic naming, and refining the question wording to explicitly guide mathematical reasoning, namely: “Estimate the

year in which 50% of Jakarta’s area will be inundated by seawater and explain your reasoning mathematically.” These findings indicate that despite the linguistic challenges, students demonstrated emerging statistical reasoning: they were able to read data representations, identify relevant patterns, and attempt data-based predictions, as evidenced by their written responses showing recognition of trends across time (see Figure 4), which confirms that the task is capable of eliciting statistical reasoning even at the readability refinement stage. These revisions were made to improve language clarity and ensure that students’ predictions were supported by data-based reasoning. The revised task was finalized as Prototype II.

Small group

Following the revisions made to Prototype I based on feedback from the expert review and one-to-one evaluation stages, Prototype II was developed and subsequently tested in the small group stage. This stage aimed to examine the practicality of the task, particularly its clarity, usability, and feasibility when implemented with multiple students in a classroom-like setting.

Prototype II was administered to nine junior high school students with varied levels of mathematical ability. Students were given two class period (equivalent to 70 minutes) to complete the task. During the implementation, the researcher closely observed students’ interactions with the task, noted questions raised by students, and conducted brief interviews to collect their comments, suggestions, and perceived difficulties while solving the problem.

Overall, most students were able to understand the task instructions and engage with the climate change context appropriately. Students demonstrated the ability to read the provided data, recognize patterns, and attempt to justify their predictions. These observations indicate that the task

was generally practical and could be implemented within the allocated instructional time. However, several students encountered difficulties in producing accurate solutions, which were mainly related to limited prior knowledge and unfamiliarity with PISA-like problem formats rather than issues with task clarity or language.

Based on these findings, the researcher concluded that no further revisions were necessary at this stage. The difficulties observed were considered reasonable given students’ prior learning experiences and did not indicate weaknesses in the task design. Therefore, Prototype II was retained in its current form for the subsequent evaluation stage.

Table 7 presents the results of the practicality questionnaire administered during the small group evaluation, showing that the developed task achieved an average practicality score of 78.8%, which falls into the “Practical” category according to the established criteria ³⁵. This finding indicates that the PISA-like task was well received by students in terms of content presentation, clarity of language, and functional usability, confirming that Prototype II can be effectively implemented in classroom settings and is suitable for use in the subsequent evaluation phase.

The field test stage was conducted to examine the potential effect of the developed PISA-like tasks on students’ statistical literacy in an authentic classroom setting. At this stage, the focus was not on measuring learning gains but on identifying whether the tasks could elicit students’ statistical literacy as intended within the Uncertainty and Data domain.

The field test was implemented in Grade IX A of Secondary School 6 Kota Jambi. Students worked individually on the Final Prototype, which had undergone expert review, one-to-one, and small group evaluations. Students’ written responses were analyzed qualitatively using three indicators of statistical literacy: Problem Understanding (PU), Data Processing (DP), and Data Interpretation (DI). To illustrate the emergence of these indicators, responses from two representative students, coded RAM and ASFZ, are presented. These students were selected on the basis of the clarity and contrasting depth of statistical reasoning demonstrated in their written responses rather than by achievement category, as the analytical focus of this stage was to illustrate how the tasks elicit statistical literacy indicators across varying reasoning patterns.

The Problem Understanding (PU) indicator refers to students’ ability to comprehend the problem situation, identify relevant information, and recognize what is being asked. Figure 5 presents RAM’s response to the Jakarta is singking task. The student correctly identified that the problem required predicting the year in which 50% of Jakarta’s area would be inundated. RAM recognized that the prediction should be based on the trend shown in the data, indicating an

understanding that the task involved analyzing changes over time rather than describing isolated values. This understanding was confirmed through a short interview:

This response shows that RAM clearly understood both the context and the objective of the task. The PU indicator emerged clearly, indicating that the task successfully guided students to understand the problem context and the nature of the required prediction.

Figure 6 presents ASFZ's response to the same task. ASFZ demonstrated an understanding of the concept being asked, namely calculating the average rate of increase and applying a linear approach to predict the year in which the percentage of inundated area would reach 50%. Although the numerical values used were not fully consistent with the data in the figure, the reasoning steps demonstrated were already appropriate. This was confirmed through interview:

This response shows that ASFZ understood the basic modeling steps required for making a prediction. The PU indicator emerged very clearly on ASFZ, indicating a strong grasp of both what the task demanded and how data should be used to address it.

The Data Processing (DP) indicator focuses on students’ ability to process numerical information, select strategies, and perform calculations or estimations to support their reasoning. Figure 7 shows RAM’s written work when attempting to determine the predicted year. The student identified an increasing pattern in the data and attempted to extend this pattern to reach the 50% threshold. However, RAM relied on informal estimation rather than a systematic calculation of the rate of increase, resulting in an inaccurate prediction.This reasoning was clarified during the interview:

Although the calculation was not fully accurate, RAM engaged in data processing by using the available data to make an estimate rather than guessing randomly. The DP indicator appeared in the student’s response, showing engagement with data-based reasoning, even though the applied strategy was still intuitive and not fully structured.

Figure 8 shows ASFZ's written work for the same task. ASFZ demonstrated a substantially more systematic approach, calculating the total increase of approximately 30% over a 50-year span, dividing it to obtain an annual rate of increase, and then applying linear extrapolation to predict when the inundated area would reach 50%. The error that occurred was only in the selection of the initial value, not in the data processing procedure itself. This was confirmed through interview:

Based on both the written response and the interview, ASFZ carried out the data processing in a sequential and logical manner. The DP indicator emerged strongly, reflecting a structured mathematical approach to prediction under uncertainty.

The Data Interpretation (DI) indicator evaluates students’ ability to draw conclusions, justify predictions, and relate results to the real-world context. Figure 9 presents RAM’s conclusion. The student provided a predicted year and justified it by referring to the observed increase in inundated area. While the explanation was brief and not mathematically rigorous, RAM acknowledged that the prediction was an estimate based on data trends rather than an exact value. This was reflected in the interview response:

This explanation demonstrates an emerging ability to interpret data and communicate conclusions within a contextual problem. The DI indicator was evident, as the student was able to connect numerical trends to a real-world conclusion, despite limitations in justification depth.

Figure 10 presents ASFZ's conclusion for the same task. ASFZ performed a logical and systematic interpretation, applying an average rate of increase and linear prediction to arrive at a specific projected year. Although the final numerical result differed from the correct answer due to an incorrect starting value, the interpretive process itself was sound and well-reasoned. This was confirmed through interview:

This response demonstrates that ASFZ's interpretive process was strong, even though the final result was not fully accurate. The DI indicator emerged clearly, with ASFZ able to connect the

calculated rate to a contextually meaningful conclusion about Jakarta's projected inundation. Taken together, the contrasting responses of RAM and ASFZ illustrate that the developed task has a positive potential effect in eliciting varying levels of statistical literacy, while also revealing the need for instructional scaffolding to support more rigorous and complete written interpretation.

The analysis conducted during the field test focused on identifying representative patterns of students’ responses and illustrating how the developed tasks elicited problem understanding, data processing, and data interpretation.

Building upon the field test findings, the assessment phase aimed to examine the overall pattern of students' statistical literacy by analyzing all 37 students' written responses to identify the potential effects of the developed tasks. Consistent with the design research approach, this phase did not aim to measure learning gains, but to analyze how students' reasoning emerged when engaging with the tasks in an authentic classroom setting. Students' written responses were analyzed qualitatively using three indicators of statistical literacy Problem Understanding (PU), Data Processing (DP), and Data Interpretation (DI) treated as analytical categories to capture students' reasoning processes rather than as summative performance measures. The overall results of the analysis are summarized in Table 8.

As shown in Table 8, all three indicators fell within the moderate category, with PU achieving the highest score (76.06%) and DI the lowest (68.69%). Regarding Problem Understanding, most students were able to recognize the problem context, identify relevant information from graphs, tables, and maps, and distinguish between historical and predicted data, although some understood the context only partially by citing numerical values without explaining their meaning in relation to the problem. Regarding Data Processing, students engaged in varied data operations beyond routine calculation, including computing differences between periods, identifying trends, and calculating average rates of change to make predictions, though this engagement was not uniform as some students performed calculations correctly but could not articulate their steps in writing, while others understood the computational logic but made numerical errors. Regarding Data Interpretation, this indicator showed the greatest variation, with some students drawing well-grounded conclusions connected to real-world climate phenomena while others offered overly general or intuition-based interpretations without adequate data support. Overall, these findings indicate that the developed tasks have a positive potential effect in eliciting all three key aspects of statistical literacy, while also revealing the need for instructional scaffolding to support deeper interpretation and more rigorous written justification when working with data involving uncertainty.