Learning through Simulations: The Ship Simulator for Learning the Rules of the Road

Learning through Simulations: The Ship Simulator for Learning the Rules of the Road

Abstract—The user study presented in this paper has investigated the potential benefits of incorporating simulation into the learning process. Simulation-based learning has been employed in the field of science and technology to improve the quality of teaching and learning. How effective is simulation in the enhancement of learning, especially compared with traditional learning techniques, is an important research question that can lead to significant benefits. The work presented in this paper has examined the differences between two instructional techniques: using simulation, and respectively, studying online material. For this purpose, we used a simulation that helps students learn about the rules of ship navigation, with the hypothesis that simulation is better than online study for learning difficult concepts. The study used simulation and online material in a self-study environment, with the research aimed at identifying the best of the two learning methods. The results show that the participants who used simulation during their self-study increased their knowledge from pretest to post test. These results also suggest that, compared with traditional learning techniques, integrating simulation into education would be a better approach to enhance learning.

Keywords—user study; simulation; education; learning methods; instructional techniques; rules of ship navigation.

 I. INTRODUCTION

Simulations are instructional scenarios that could be integrated into education, particularly because through them students experience a stronger sense of reality. In general, traditional instructional approaches such as classroom-based learning or reading from textbooks can be challenging. Through traditional education the students learn a diversity of subjects but they may not know how to apply their gained knowledge to real-world problems. An alternative to the traditional learning techniques is the introduction of computer simulations for instructional purposes. In [1], the authors determined that when simulations are incorporated into the education they have a positive influence on the learning outcomes. They facilitate students to acquire knowledge through direct experience, especially if the simulations are utilized as attractive, easy to understand interactive media. Simulation-based learning has the capability to incorporate real world training scenarios that otherwise are strenuous to simulate in a traditional teaching environment. Thus, it can help students sustain their attention throughout extended learning periods.

The main objective of simulation-based learning is that students learn while doing or experiencing things in the simulated world, which resembles the real world. The integration of effective features of simulations into the education can enhance learning. More effective training systems are vital to the survival of education because they can help students learn a variety of subjects actively rather than passively.

Many real world situations and problems could be represented in simulation-based systems, which can be useful for the students to get a better feel of practical experiences. Software tools for creating simulation environments are becoming more advanced and easier to employ [2].

Although simulations can provide enhanced learning environments, some disadvantages include: (a) students need training before using simulation tools; (b) teaching methods and the students’ learning may not match; (c) there is no instant feedback provided to the user; (d) before the students use the simulation environment, it is necessary that they have basic computer skills and also know the various components of the simulation [2].

A. Rules of the Road Ship Simulator

The Rules of the Road Ship Simulator (RoRSS) we created part of our research work provides a platform for the students to learn about ship navigation rules by facilitating interactive real-world experiences. Figure 1 shows a screenshot taken during a practice session with RoRSS. It is worth noting that RoRSS has two main components or sections: a practice section and a quiz section.

In the practice section, the students learn about the ship’s type, target angle, light configurations, and locations of ships, thereby they are acquiring the concepts and information needed for safe-navigation on the open ocean. Navigation is the maneuvering of a ship from one point to another on the open ocean, similar to driving the car on the road [3]. Safe ship navigation requires knowledge about ship types, shapes, lights, light configurations, target angles, and the associated rules of the road to control the vessel and avoid collisions between ships. So far we have incorporated in RoRSS information about 10 ships used by the US Navy.

In the practice section, the students can use the right and left arrow keys to rotate and maneuver the ship around. The target angle and view of the ship given at the top-center of the screen enable the users to know about the direction to which the target ship is headed. Users can use the buttons Night, Dusk, and Day to switch between the night, dusk, and daytime views of the ship. With the system’s night option available in the practice section, the students can understand different light configurations: Underway, Not Under Command, Making Way, Not Making Way, Restricted in Ability to Maneuver,

Constrained by Draft, Pushing Ahead, Towing Astern, Engaged

Fig. 1. Screenshot from a practice session with RoR

in Fishing, Engaged in Sailing, and At Anchor. For example, Underway means not attached to the ground but not necessarily moving through the water while Making Way always means moving through the water [3]. The users can also view other ships at different distances by using the Near, Medium, and Far buttons that are provided in the simulator’s GUI. To avoid collisions, one must know the activities and positions of the ships as well as all possible light configurations during the night.

B. Problem Statement

When traditional learning methods are used, students extract knowledge from textbooks, classroom lectures, labs and tutorials, or from material available online. However, it has been shown that this way of learning has several limitations, especially in terms of students applying their newly acquired knowledge to actual real-world scenarios [4]. This lag between the learning process and the application of the learned knowledge manifests in students having difficulty to use new principles and concepts in practice. Therefore, this problem merits investigation to determine whether other learning methods, such as simulation-based learning, could be more effective in this respect.

C. Purpose of the Research

The purpose of the study is to compare the effects on student performance of simulation-based learning versus traditional studying of online material. Our research study employs RoRSS as an environment for students to self-study the rules of ship navigation. In particular, the research explores the effectiveness of simulation in learning concepts and information related to safe navigation, including ship types, shapes, lights, light configurations, and target angles. The design method applied involves qualitative and quantitative analyses of data. The following is the research question we are trying to address in this paper: Can we improve student learning by using a simulation-based approach over employing a traditional learning method such as the studying of online material? The sub-questions are: Do users perform better (as reflected in quiz scores) when learning using a simulation environment rather than when using a traditional learning method? Do users learn a given set of concepts in less time when using simulation versus when studying online material?

The null hypothesis is:

H0: The performance achieved and time taken by students to learn ship navigation concepts and information using a simulation environment do not significantly differ from the performance achieved and time taken by students who use online study material.

The alternate hypothesis is:

H1: The performance achieved and time taken by students to learn ship navigation concepts and information using a simulation environment are different from the performance achieved and time taken by students who use online study material.

The remaining of this paper is organized as follows: Section 2 covers the related work, followed by Section 3 which describes the methodology applied. Results are presented and discussed in Section 4 while Section 5 presents the conclusions of the paper and outlines directions of future work.

 II. RELATED WORK

Integrating technology into education has become popular nowadays. In particular, simulation has been employed as a

teaching tool to provide students with a realistic experience. In their paper [2], Mohammed et al. have discussed the impact of simulation-based teaching on students’ learning outcomes compared with the hands-on approach in the lab-based environment. The results of their study revealed that simulation is effective in promoting learning when it is combined with hands-on activity. In [5], Sulaiman et al. address the effectiveness of simulation on students’ academic performance by comparing a simulation-based technique with a lecture based method of teaching. The study found that the students taught with the simulation game technique performed better than the students who used the lecture method for learning. According to Veenman, Elshout and Busato [6], simulations that address problem-oriented concepts improve cognitive abilities, creativity, and problem-solving capabilities of the students. Simulation-based learning helps students develop critical and strategic thinking skills as well as enables them to learn interactively various theories and concepts. Thus, simulations promote active learning. The main advantage of simulationbased training systems is that they provide a strong tool for students to develop strategic planning and thinking.

Simulations are not only used in education but also in other areas such as business, medical, transportation, and search and rescue operations. In [7], Siddiqui et al. address the potential advantage of simulation for disaster operations. They investigated the effect of a simulation environment for Unmanned Aerial Vehicles (UAVs) on training personnel who can experience real-world control of UAVs for rescue operations. In [8], Shin examined the effectiveness of patient simulation in nursing education through meta-analysis of several primary studies. These studies focused on the effects of simulation on patients in nursing. The authors identified significant postintervention betterment among participants who received simulation education compared to the control group, thereby concluded that simulation is more potent educational tool than the traditional learning method [8].

In [9], the authors investigated the effect of an online simulation-based learning system on students performance in learning the concepts of linked-list structures. Their results exhibited that the participants considered simulation as a useful learning environment. Furthermore, simulations help learners enhance their skills through practice and prepare them to more easily understand complex concepts. In particular, simulation enhances learning by inclusion of tasks, problems, or cases [10]. A study was conducted in Osun-State, Nigeria, to identify how simulations and game environments aid in increasing student motivation and improving their attitude towards learning mathematics [11]. The results of their study showed that the students were positively affected by the use of games and simulations. In [12], Roger and Michelene investigated the difference between learning using computer simulations and reading from text. They have used the domain topic project management for their study. The students in the experimental group were learning from simulation about project management activities such as interpreting the relationship between project management decisions and measure project outcomes. The control group focused on learning different sections about project management by reading related texts. Their results showed that computer simulations enhance the learning of implicit domain knowledge from pre-test to post-test.

In [13], the author examined the impact of an interactive computer simulation to foster student learning outcomes. The results indicated that both observation and experimentation interactivity have an equal effect on facilitating students performance. Moreover, the results also showed that when compared with structured prompt scaffolding, the driving questions scaffolding helped the students to learn the concepts of sinking and floating better. Computer simulations act as a training tool to reinforce students model-based reasoning [14].

 III. METHODOLOGY

We studied the effectiveness of using RoRSS in learning the concepts of ship navigation rules. To this end, the control group of students learned these concepts by self-studying related online material while the experimental (intervention) group of students learned using the ship simulator. The results of the study were based on the students’ scores and quiz completion times, for both the simulation and the online study material approaches. Part of this user study, the responses from 44 participants were examined quantitatively and qualitatively.

A. Participants

The participants for this user study were university graduate and undergraduate students with very limited knowledge of the domain (ship navigation rules). 44 participants with no prior experience in our simulation-based learning system were recruited by word of mouth. Among 44 participants, there were 22 participants in the intervention group and the other 22 participants were considered to be the control group. The participants ages ranged from 18 to 35 years. Users included both genders, and there was an equal percentage of males and females in the experimental group.

B. Apparatus

The platform used for simulation was Windows. The hardware used consisted of an Intel core i7 processor and the total memory of 16 GB. The application was developed using the Unity game engine with C as the programming language. The data from the user study was stored in a SQLite database which has been integrated into the Unity game engine.

C. Procedure

The study was approved by our university’s IRB (Institutional Review Board) office and the experiment was conducted in the university library. Prior to the experiment, the participants were provided with a consent form that explained the objective of the research and the procedure of the experiment. The participation in this study was on a voluntary basis. Participants were directed to the further steps only if they agreed to be in the study. The treatment group used the RoRSS, whereas the control group used online study material.

The online study material provided to the control group was the abbreviated guide to navigation rules of the road available from [15]. The control group as well as the intervention group first completed a pre-test survey which included demographic data, their current knowledge about ship navigation rules, and their most preferred learning method. Due to the complexity of the subject matter, both groups were allowed to get a brief idea about ship navigation by reading a related online study material for 5 minutes.

Afterwards, they were asked to take a pre-test. The pretest given was a quiz that contained 20 questions. After the quiz, users underwent a self-study session. The intervention group used RoRSS’ practice section, whereas the control group was presented with the online material ”Abbreviated guide to navigation rules of the road” [15] for their self-study session. The time allotted for both groups during their self-study was 20 minutes. The self-study session was followed by another quiz (post-test) consisting of 20 questions with the same level of difficulty as that of the pre-test. For consistency, every quiz provided to the users was generated automatically. Snapshots taken during the RoRSS quiz session are shown in Figures 2, 3, and 4.

Figure 2 displays one of the ships included in the simulator. The students had to identify the type of ship displayed by choosing one of three possible answers. Once they finished answering the question, the students were able to proceed to the following question by clicking on the Next Question button.

Figure 3 shows a ship’s light configuration during the night. At night, lights are the only option to identify a ship and its target angle. During the quiz, the students were asked to identify the particular light configuration of the ship.

Figure 4 displays the ship heading towards a specific direction. Students had to select the target angle of the displayed ship. To do that, they had to move the needle of the compass to select the estimated target angle.

The quiz score and the time took for each question were recorded in the SQLite database. Finally, a post-test survey was given to the participants to help analyze whether they liked the system and whether they prefer to use the simulation feature or other type of learning. In addition, their comments about the simulation were also collected. For each participant, the whole experiment lasted approximately 40 minutes.

Fig. 2. Simulator quiz question regarding the type of a ship

Fig. 3. Simulator quiz question regarding the light configuration of a ship

D. Design

The experiment was a 2 x 2 in-between subjects design. There were two independent variables and two dependent variables.

Independent variables: Simulation, online study material

Dependent variables: Score, quiz completion time

For the design, we used four methods of evaluation. In the first method, we assessed the pre-test and the post-test scores of both the intervention and control groups. We analyzed the scores as a criterion to evaluate the students’ improvement in learning. In the second method, pre-test and post-test quiz completion times were analyzed for both the intervention and control groups.

In the third method, we compared the delta, or change, in the pre-test and post-test quiz scores of the students who used RoRSS’ practice section with that of the students who learned using online study material. Finally, in the fourth method, we compared the delta in the pre-test and post-test quiz completion times between the two groups.

The quantitative parameters we considered were quiz score and quiz completion time. The qualitative parameter considered was the usefulness of RoRSS’ practice section in learning concepts and information pertaining to the rules of ship navigation.

Fig. 4. Simulator quiz question regarding the target angle of a ship

 IV. RESULTS AND DISCUSSION

The first phase of the analysis consisted of collecting data based on the quiz attempted by the participants. The IBM SPSS Statistics Data Editor tool was used for analyzing data. The Paired-Samples T-test was used for testing within-subject factors and the Mixed-Design ANOVA (Split-Plot ANOVA) was used for testing between subject factors.

The quantitative analysis of the data collected from the user study is presented next.

Table I shows the analysis of quiz score within subjects design. For the experimental group, there was a significant improvement from pre-test (Mean = 0.3341, Std. Deviation = 0.0931) to post-test (Mean = 0.5386 , Std. deviation = 0.1185), with a p-value less than ¡ 0.5. This result thus provided strong evidence that the simulation-based learning helped the students to improve their scores.

However, for the control group who used online study material for their self-study session, there was no significant improvements from pre-test (Mean = 0.3182 , Std. deviation = 0.1006 ) to post-test (Mean = 0.3545, Std. deviation = 0.0987), with a p-value =0.73 (p¿0.05). The result thus indicates that the students were not able to improve their scores when they used online study material for their self-study session.

Table 1: Analysis of quiz scores within subjects

Table II shows the analysis of quiz times within subjects design. From the details given in Table II, it can be seen that for the experimental group there was a slight decrease in mean time from pre-test (Mean = 11.9659 , Std. deviation =

2.8421) to post-test (Mean = 10.9327, Std. deviation = 3.2428). However, the p-value obtained was p = .118 (p ¿ .05) indicating that there is no statistical significance difference even though they used simulation for learning.

Based on the delta mean values presented in Table I and II, Table III shows the differences in scores between subjects: the experimental group and the control group. Results from the mixed-design ANOVA test indicate that there is a significant difference in the pre-test and post-test scores of both the control and the experimental groups. This is based on the fact that the calculated p-value is less than 0.05 and the F-value is 822.079. Therefore, the null hypothesis was rejected. Similarly, the difference in times taken for the quiz between subjects is also statistically significant because the p-value is less than .05. From Table III, it can be implied that the simulation-based approach improved the outcomes of the students’ learning when compared with the outcomes of the online material study approach.

Table 2: Analysis between subjects

Figure 5 shows the graph of the scores between subjects. The x-axis represents when the quiz was taken (1 = pre-test and 2 = post-test) and the y-axis represents the estimated marginal means of the scores. The graph indicates that there is only a small difference (0.15 vs 0.36) in scores between pre-test and post-test for the control group. In contrast, the experimental group’s scores increased substantially from pretest to the post-test (0.27 vs 0.54). This indicates that the performance of the students who used simulation for learning increased substantially.

Figure 6 shows the graph of the time taken for the quiz between subjects. The x-axis represents when the quiz was taken (1 = pre-test and 2 = post-test) and the y-axis represents the estimated marginal means of the times taken for the quiz. The graph indicates that the time taken for the quiz by the experimental group decreased significantly (the marginal mean decreased from 12.0 to 11.0) whereas the control group has a slight increase in the time taken from pre-test to post-test (the marginal mean increased from 2.5 to 5.0).

Fig 6. Online vs Simulation in terms of pre-test and post-test quiz completion times

Overall, the results from the quantitative analysis demonstrate that the simulation is a more effective tool for learning Fig 7. Students’ most preferred learning method before

Fig 8. Students’ most preferred learning method after learning using RoRSS

Figure 9 shows the number of experimental and control group participants’ responses regarding the level of difficulty of learning during their self-study sessions. It can be seen that the highest percentage (55%) of the experimental group characterized their difficulty level of learning using simulation as moderate. On the other hand, only 45% of the control group indicated their response as moderate. Out of 22 experimental group participants, 19 favored the use of simulation whereas 3 found the simulation difficult to use. In the case of self-study using online material, 12 of the 22 control group participants described their level of learning as difficult or very difficult. Moreover, none of the control group participants found the Fig 10. The user experience with the RoRSS simulation

Figure 11 and Figure 12 show the responses of the students to the post-survey questionnaire that complimented the research questions. The students were asked to opt from strongly agree to strongly disagree in response to the statements: (i) I understand more about ship types, target angle and light configuration after self-study using simulation-based tutoring system (Figure 11) and (ii) I am likely to recommend learning using simulation to others (Figure 12). As the two pie charts portray, over 90% of the students had the opinion that the use of simulation helped them in understanding concepts and information pertaining to ship navigation and over 80% were likely to recommend simulation-based learning to others.

Fig 12. Responses to the statement ”I am likely to

recommend learning using simulation to others”

Figure 13 shows the students’ responses regarding their level of knowledge about ship navigation before and after selfstudy using simulation. A glance at the graph enlightens us of the fact that the students were able to learn more by using simulation-based learning. More than 70% of the students gave positive feedback about the simulation-based learning method. Only 22% of the students in the experimental group did not find the simulation as useful.

Fig 13. Level of knowledge about ship navigation before and after self-study using simulation

Figure 14 shows the students’ responses regarding their

 V. CONCLUSION AND FUTURE WORK

This paper has presented a user study that investigated the effects of a simulation-based approach on the students’ learning of the concepts pertaining to ship navigation. Based on the findings of the study, it is concluded that the users who used simulation achieved significantly better results on the post-test quiz, whereas the users who used online study material did not perform well.

The intended users of this simulation are midshipmen where they can experience a realistic environment for learning about ship types, shapes, lights, light configurations, and target angles. The domain of this particular simulation (ship navigation) is complex for the regular students, who largely are not familiar with it. The important finding is that the simulationbased learning significantly improved the experimental group’s performance over time, despite the fact that the domain is complex and the participants are not the intended users of the simulation. The performance-based evaluation provided evidence that with the help from simulation, students could improve their post-test quiz scores. This work thus demonstrated that a simulation-based approach could enhance the learning outcomes when compared with a traditional educational method. Furthermore, the results of the qualitative analysis showed that the students are interested and willing to use the simulation as a learning tool in the future.

For future work, we are interested in investigating the students’ performance using a Bayesian network approach. To this end RoRSS’ Feedback GUI will be further modified so that the students could have a better understanding of their ongoing performance during the quiz. We also plan to integrate a gamebased approach to the simulation so that learning could also be fun.

REFERENCES

[1]      D. Vlachopoulos and A. Makri, The effect of games and simulations on higher education: a systematic literature review International Journal of Educational Technology in Higher Education, vol. 14, pp. 1-33, 2017.

[2]      T. MOHAMMED, Impact of Simulation-based and Hands-on Teaching Methodologies on Students Learning in an Engineering Technology Program, In Proceedings of 121st ASEE Annual Conference and Exposition, Indianapolis, IN, 2014.

[3]      R. B Michael, and A. C. Joseph, COLREGS – Based Navigation of Autonomous Marine Vehicles, Proceedings of the 2004 IEEE/OES Autonomous Underwater Vehicles, Sebasco, ME, USA, 1718 June 2004. pp. 32-39.

[4]      C. Raymond, Do Role-Playing Simulations Generate Measurable and Meaningful Outcomes? A Simulations Effect on Exam Scores and Teaching Evaluations, International Studies Perspectives, vol. 11, pp. 51-60, 2010.

[5]      B. Sulaiman, I. B. Mustapha, and B. B. Ibrahim, Effect of Simulation Techniques and Lecture Method on Students Academic Performance in Mafoni Day Secondary School Maiduguri, Borno State, Nigeria, Journal of Education and Practice, vol. 7, pp. 113-117, 2016.

[6]      M. V. Veenman, J. Elshout, and V. Busato, Metacognitive mediation in learning with computer-based simulations, Journal of Computers in Human Behavior, vol. 10, pp. 93-106, 1994.

[7]      K. T. A. Siddiqui, D. Feil-Seifer, T. Jiang, S. Jose, S. Liu, and S. Louis, Development of a Swarm UAV Simulator Integrating Realistic Motion Control Models For Disaster Operations, Proceedings of the ASME Dynamic Systems and Controls Conference, Tysons Corner, Virginia, October, October, 2017, pp. 1-10.

[8]      S. Shin, J. H. Park and J. H. Kim, Effectiveness of patient simulation in nursing education: Meta-analysis, Nurse Education Today 35(1), 176182, 2015.

[9]      A. Lai, T. Wu, G. Lee, and H. Lai, Developing a web-based simulationbased learning system for enhancing concepts of linked-list structures in Data Structures curriculum, Proceedings of the 3rd International Conference on Artificial Intelligence, Modelling and Simulation, 2015, pp. 185-188.

[10]   C. Teresa, Using Simulations to Enhance Teaching and Learning: Encouraging the Creative Process , Educational Researcher, VSTE Virginia Society for technology in Education Journal, vol. 21, pp. 1-7, 2006.

[11]   M.K. Akinsola, The effect of simulation-games environment on students achievement and attitudes to mathematics in secondary schools, The Turkish Online Journal of Educational Technology, TOJET, vol. 6, pp. 1-7, 2007.

[12]   R.S. Taylor and M.T.H. Chi, Simulation versus text: Acquisition of implicit and explicit information, Journal of Educational Computing Research, vol. 35, pp. 289-313, 2006.

[13]   H. Chang, How to augment the learning impact of computer simulation? The designs and effects of interactivity and scaffolding, Journal of Interactive Learning Environment, vol. 25, pp. 1083-1097, 2017.

[14]   M. Develaki, Using computer simulations for promoting model-based reasoning, Science and Education, vol. 26, pp 1001-1027, 2017.

[15]   QUICK REFERENCE Abbreviated Guide To Navigation Rules Of the Road Based on the Navigation Rules International Inland (Commandant Instruction M16672.2D). (1999) [Online]. Available: https://www.uscg.mil/hq/cgcvc/cvc3/references/Rules of Road Quick R eference.pdf